Hearing illusory sounds in noise: the timing of sensory-perceptual transformations in auditory cortex

Theta and alpha oscillatory signatures of auditory sensory and cognitive loads during complex listening

The neuronal signatures of sensory and cognitive load provide access to brain activities related to complex listening situations. Sensory and cognitive loads are typically reflected in measures like response time (RT) and event-related potentials (ERPs) components. It's, however, strenuous to distinguish the underlying brain processes solely from these measures. In this study, along with RT- and ERP-analysis, we performed time-frequency analysis and source localization of oscillatory activity in participants performing two different auditory tasks with varying degrees of complexity and related them to sensory and cognitive load. We studied neuronal oscillatory activity in both periods before the behavioral response (pre-response) and after it (post-response). Robust oscillatory activities were found in both periods and were differentially affected by sensory and cognitive load. Oscillatory activity under sensory load was characterized by decrease in pre-response (early) theta activity and increased alpha activity. Oscillatory activity under cognitive load was characterized by increased theta activity, mainly in post-response (late) time. Furthermore, source localization revealed specific brain regions responsible for processing these loads, such as temporal and frontal lobe, cingulate cortex and precuneus. The results provide evidence that in complex listening situations, the brain processes sensory and cognitive loads differently. These neural processes have specific oscillatory signatures and are long lasting, extending beyond the behavioral response.

Evidence for predictions established by phantom sound

Predictions, the bridge between the internal and external worlds, are established by prior experience and updated by sensory stimuli. Responses to omitted but unexpected stimuli, known as omission responses, can break the one-to-one mapping of stimulus-response and can expose predictions established by the preceding stimulus built up. While research into exogenous predictions (driven by external stimuli) is often reported, that into endogenous predictions (driven by internal percepts) is rarely available in the literature. Here, we report evidence for endogenous predictions established by the Zwicker tone illusion, a phantom pure-tone-like auditory percept following notch noises. We found that MMN, P300, and theta oscillations could be recorded using an omission paradigm in subjects who can perceive Zwicker tone illusions, but could not in those who cannot. The MMN and P300 responses relied on attention, but theta oscillations did not. In-depth analysis shows that an increase in single-trial theta power, including total and induced theta, with the endogenous prediction, is lateralized to the left frontal brain areas. Our study depicts that the brain automatically analyzes internal perception, progressively establishes predictions and yields prediction errors in the left frontal region when a violation occurs.

Expectations boost the reconstruction of auditory features from electrophysiological responses to noisy speech

Online speech processing imposes significant computational demands on the listening brain, the underlying mechanisms of which remain poorly understood. Here, we exploit the perceptual "pop-out" phenomenon (i.e. the dramatic improvement of speech intelligibility after receiving information about speech content) to investigate the neurophysiological effects of prior expectations on degraded speech comprehension. We recorded electroencephalography (EEG) and pupillometry from 21 adults while they rated the clarity of noise-vocoded and sine-wave synthesized sentences. Pop-out was reliably elicited following visual presentation of the corresponding written sentence, but not following incongruent or neutral text. Pop-out was associated with improved reconstruction of the acoustic stimulus envelope from low-frequency EEG activity, implying that improvements in perceptual clarity were mediated via top-down signals that enhanced the quality of cortical speech representations. Spectral analysis further revealed that pop-out was accompanied by a reduction in theta-band power, consistent with predictive coding accounts of acoustic filling-in and incremental sentence processing. Moreover, delta-band power, alpha-band power, and pupil diameter were all increased following the provision of any written sentence information, irrespective of content. Together, these findings reveal distinctive profiles of neurophysiological activity that differentiate the content-specific processes associated with degraded speech comprehension from the context-specific processes invoked under adverse listening conditions.

Similar EEG Activity Patterns During Experimentally-Induced Auditory Illusions and Veridical Perceptions

Hallucinations and illusions are two instances of perceptual experiences illustrating how perception might diverge from external sensory stimulations and be generated or altered based on internal brain states. The occurrence of these phenomena is not constrained to patient populations. Similar experiences can be elicited in healthy subjects by means of suitable experimental procedures. Studying the neural mechanisms underlying these experiences not only has the potential to expand our understanding of the brain's perceptual machinery but also of how it might get impaired. In the current study, we employed an auditory signal detection task to induce auditory illusions by presenting speech snippets at near detection threshold intensity embedded in noise. We investigated the neural correlates of auditory false perceptions by examining the EEG activity preceding the responses in speech absent (false alarm, FA) trials and comparing them to speech present (hit) trials. The results of the comparison of event-related potentials (ERPs) in the activation period vs. baseline revealed the presence of an early negativity (EN) and a late positivity (LP) similar in both hits and FAs, which were absent in misses, correct rejections (CR) and control button presses (BPs). We postulate that the EN and the LP might represent the auditory awareness negativity (AAN) and centro-parietal positivity (CPP) or P300, respectively. The event-related spectral perturbations (ERSPs) exhibited a common power enhancement in low frequencies (<4 Hz) in hits and FAs. The low-frequency power enhancement has been frequently shown to be accompanied with P300 as well as separately being a marker of perceptual awareness, referred to as slow cortical potentials (SCP). Furthermore, the comparison of hits vs. FAs showed a significantly higher LP amplitude and low frequency power in hits compared to FAs. Generally, the observed patterns in the present results resembled some of the major neural correlates associated with perceptual awareness in previous studies. Our findings provide evidence that the neural correlates associated with conscious perception, can be elicited in similar ways in both presence and absence of externally presented sensory stimuli. The present findings did not reveal any pre-stimulus alpha and beta modulations distinguishing conscious vs. unconscious perceptions.

Lexical Information Guides Retuning of Neural Patterns in Perceptual Learning for Speech

A listener's interpretation of a given speech sound can vary probabilistically from moment to moment. Previous experience (i.e., the contexts in which one has encountered an ambiguous sound) can further influence the interpretation of speech, a phenomenon known as perceptual learning for speech. This study used multivoxel pattern analysis to query how neural patterns reflect perceptual learning, leveraging archival fMRI data from a lexically guided perceptual learning study conducted by Myers and Mesite [Myers, E. B., & Mesite, L. M. Neural systems underlying perceptual adjustment to non-standard speech tokens. Journal of Memory and Language, 76, 80-93, 2014]. In that study, participants first heard ambiguous /s/-/∫/ blends in either /s/-biased lexical contexts (epi_ode) or /∫/-biased contexts (refre_ing); subsequently, they performed a phonetic categorization task on tokens from an /asi/-/a∫i/ continuum. In the current work, a classifier was trained to distinguish between phonetic categorization trials in which participants heard unambiguous productions of /s/ and those in which they heard unambiguous productions of /∫/. The classifier was able to generalize this training to ambiguous tokens from the middle of the continuum on the basis of individual participants' trial-by-trial perception. We take these findings as evidence that perceptual learning for speech involves neural recalibration, such that the pattern of activation approximates the perceived category. Exploratory analyses showed that left parietal regions (supramarginal and angular gyri) and right temporal regions (superior, middle, and transverse temporal gyri) were most informative for categorization. Overall, our results inform an understanding of how moment-to-moment variability in speech perception is encoded in the brain.

Thalamic gating contributes to forward suppression in the auditory cortex

The neural mechanisms underlying forward suppression in the auditory cortex remain a puzzle. Little attention is paid to thalamic contribution despite the important fact that the thalamus gates upstreaming information to the auditory cortex. This study compared the time courses of forward suppression in the auditory thalamus, thalamocortical inputs and cortex using the two-tone stimulus paradigm. The preceding and succeeding tones were 20-ms long. Their frequency and amplitude were set at the characteristic frequency and 20 dB above the minimum threshold of given neurons, respectively. In the ventral division of the medial geniculate body of the thalamus, we found that the duration of complete forward suppression was about 75 ms and the duration of partial suppression was from 75 ms to about 300 ms after the onset of the preceding tone. We also found that during the partial suppression period, the responses to the succeeding tone were further suppressed in the primary auditory cortex. The forward suppression of thalamocortical field excitatory postsynaptic potentials was between those of thalamic and cortical neurons but much closer to that of thalamic ones. Our results indicate that early suppression in the cortex could result from complete suppression in the thalamus whereas later suppression may involve thalamocortical and intracortical circuitry. This suggests that the complete suppression that occurs in the thalamus provides the cortex with a "silence" window that could potentially benefit cortical processing and/or perception of the information carried by the preceding sound.

Low-frequency oscillations reflect aberrant tone restoration during the auditory continuity illusion in schizophrenia

Patients with schizophrenia (ScZ) often show impairments in auditory information processing. These impairments have been related to clinical symptoms, such as auditory hallucinations. Some researchers have hypothesized that aberrant low-frequency oscillations contribute to auditory information processing deficits in ScZ. A paradigm for which modulations in low-frequency oscillations are consistently found in healthy individuals is the auditory continuity illusion (ACI), in which restoration processes lead to a perceptual grouping of tone fragments and a mask, so that a physically interrupted sound is perceived as continuous. We used the ACI paradigm to test the hypothesis that low-frequency oscillations play a role in aberrant auditory information processing in patients with ScZ (N = 23). Compared with healthy control participants we found that patients with ScZ show elevated continuity illusions of interrupted, partially-masked tones. Electroencephalography data demonstrate that this elevated continuity perception is reflected by diminished 3 Hz power. This suggests that reduced low-frequency oscillations relate to elevated restoration processes in ScZ. Our findings support the hypothesis that aberrant low-frequency oscillations contribute to altered perception-related auditory information processing in ScZ.

Visual Enhancement of Relevant Speech in a 'Cocktail Party'

Lip-reading improves intelligibility in noisy acoustical environments. We hypothesized that watching mouth movements benefits speech comprehension in a 'cocktail party' by strengthening the encoding of the neural representations of the visually paired speech stream. In an audiovisual (AV) task, EEG was recorded as participants watched and listened to videos of a speaker uttering a sentence while also hearing a concurrent sentence by a speaker of the opposite gender. A key manipulation was that each audio sentence had a 200-ms segment replaced by white noise. To assess comprehension, subjects were tasked with transcribing the AV-attended sentence on randomly selected trials. In the auditory-only trials, subjects listened to the same sentences and completed the same task while watching a static picture of a speaker of either gender. Subjects directed their listening to the voice of the gender of the speaker in the video. We found that the N1 auditory-evoked potential (AEP) time-locked to white noise onsets was significantly more inhibited for the AV-attended sentences than for those of the auditorily-attended (A-attended) and AV-unattended sentences. N1 inhibition to noise onsets has been shown to index restoration of phonemic representations of degraded speech. These results underscore that attention and congruency in the AV setting help streamline the complex auditory scene, partly by reinforcing the neural representations of the visually attended stream, heightening the perception of continuity and comprehension.

Illusory sound texture reveals multi-second statistical completion in auditory scene analysis

Sound sources in the world are experienced as stable even when intermittently obscured, implying perceptual completion mechanisms that “fill in” missing sensory information. We demonstrate a filling-in phenomenon in which the brain extrapolates the statistics of background sounds (textures) over periods of several seconds when they are interrupted by another sound, producing vivid percepts of illusory texture. The effect differs from previously described completion effects in that 1) the extrapolated sound must be defined statistically given the stochastic nature of texture, and 2) the effect lasts much longer, enabling introspection and facilitating assessment of the underlying representation. Illusory texture biases subsequent texture statistic estimates indistinguishably from actual texture, suggesting that it is represented similarly to actual texture. The illusion appears to represent an inference about whether the background is likely to continue during concurrent sounds, providing a stable statistical representation of the ongoing environment despite unstable sensory evidence.

Auditory textures are sounds defined by a particular statistical distribution, e.g. as is produced by rain, or a swarm of insects. Here, the authors describe a striking perceptual illusion in which sound textures are heard to continue, even though they have in fact been replaced by white noise.

Restoration and Efficiency of the Neural Processing of Continuous Speech Are Promoted by Prior Knowledge

Sufficiently noisy listening conditions can completely mask the acoustic signal of significant parts of a sentence, and yet listeners may still report the perception of hearing the masked speech. This occurs even when the speech signal is removed entirely, if the gap is filled with stationary noise, a phenomenon known as perceptual restoration. At the neural level, however, it is unclear the extent to which the neural representation of missing extended speech sequences is similar to the dynamic neural representation of ordinary continuous speech. Using auditory magnetoencephalography (MEG), we show that stimulus reconstruction, a technique developed for use with neural representations of ordinary speech, works also for the missing speech segments replaced by noise, even when spanning several phonemes and words. The reconstruction fidelity of the missing speech, up to 25% of what would be attained if present, depends however on listeners' familiarity with the missing segment. This same familiarity also speeds up the most prominent stage of the cortical processing of ordinary speech by approximately 5 ms. Both effects disappear when listeners have no or little prior experience with the speech segment. The results are consistent with adaptive expectation mechanisms that consolidate detailed representations about speech sounds as identifiable factors assisting automatic restoration over ecologically relevant timescales.

Cross-modal phonetic encoding facilitates the McGurk illusion and phonemic restoration

In spoken language, audiovisual (AV) perception occurs when the visual modality influences encoding of acoustic features (e.g., phonetic representations) at the auditory cortex. We examined how visual speech (mouth movements) transforms phonetic representations, indexed by changes to the N1 auditory evoked potential (AEP). EEG was acquired while human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables, /ba/ and /wa/, presented in auditory-only or AV congruent or incongruent contexts or in a context in which the consonants were replaced by white noise (noise replaced). Subjects reported whether they heard "ba" or "wa." We hypothesized that the auditory N1 amplitude during illusory perception (caused by incongruent AV input, as in the McGurk illusion, or white noise-replaced consonants in CV utterances) should shift to reflect the auditory N1 characteristics of the phonemes conveyed visually (by mouth movements) as opposed to acoustically. Indeed, the N1 AEP became larger and occurred earlier when listeners experienced illusory "ba" (video /ba/, audio /wa/, heard as "ba") and vice versa when they experienced illusory "wa" (video /wa/, audio /ba/, heard as "wa"), mirroring the N1 AEP characteristics for /ba/ and /wa/ observed in natural acoustic situations (e.g., auditory-only setting). This visually mediated N1 behavior was also observed for noise-replaced CVs. Taken together, the findings suggest that information relayed by the visual modality modifies phonetic representations at the auditory cortex and that similar neural mechanisms support the McGurk illusion and visually mediated phonemic restoration. NEW & NOTEWORTHY Using a variant of the McGurk illusion experimental design (using the syllables /ba/ and /wa/), we demonstrate that lipreading influences phonetic encoding at the auditory cortex. We show that the N1 auditory evoked potential morphology shifts to resemble the N1 morphology of the syllable conveyed visually. We also show similar N1 shifts when the consonants are replaced by white noise, suggesting that the McGurk illusion and the visually mediated phonemic restoration rely on common mechanisms.

Universal Phonological Features

The Theory of universal grammar suggests that human languages may share some similarities at the phonological level. Based on this hypothesis, we further propose a language generalization effect (LGE) and hypothesize that people may inherit the universal phonological features from their native languages and then transfer them to foreign languages. To test this hypothesis, in two experiments, participants listened to a pair of normal and syllable reversed recordings (Experiments 1a-1d) or normal and phonemic reversed recordings (Experiments 2a-2d) in unknown and native languages and reported their similarities. The results indicated that participants were more sensitive to the dissimilarities of normal recordings than to reversed ones. These results suggest that participants could identify the universal phonological features from normal recordings, but not reversed ones, and then adopt these features to better verify the dissimilarities of normal recordings. In sum, the current study first proposed the LGE and reported primary evidence to support it.

Reduced low-frequency power and phase locking reflect restoration in the auditory continuity illusion

Interruptions in auditory input can be perceptually restored if they coincide with a masking sound, resulting in a continuity illusion. Previous studies have shown that this continuity illusion is associated with reduced low-frequency neural oscillations in the auditory cortex. However, the precise contribution of oscillatory amplitude changes and phase alignment to auditory restoration remains unclear. Using electroencephalography, we investigated induced power changes and phase locking in response to 3 Hz amplitude-modulated tones during the interval of an interrupting noise. We experimentally manipulated both the physical continuity of the tone (continuous vs. interrupted) and the masking potential of the noise (notched vs. full). We observed an attenuation of 3 Hz power during continuity illusions in comparison with both continuous tones and veridically perceived interrupted tones. This illusion-related suppression of low-frequency oscillations likely reflects a blurring of auditory object boundaries that supports continuity perception. We further observed increased 3 Hz phase locking during fully masked continuous tones compared with the other conditions. This low-frequency phase alignment may reflect the neural registration of the interrupting noise as a newly appearing object, whereas during continuity illusions, a spectral portion of this noise is delegated to filling the interruption. Taken together, our findings suggest that the suppression of slow cortical oscillations in both the power and phase domains supports perceptual restoration of interruptions in auditory input.

Dynamic cortical representations of perceptual filling-in for missing acoustic rhythm

In the phenomenon of perceptual filling-in, missing sensory information can be reconstructed via interpolation or extrapolation from adjacent contextual cues by what is necessarily an endogenous, not yet well understood, neural process. In this investigation, sound stimuli were chosen to allow observation of fixed cortical oscillations driven by contextual (but missing) sensory input, thus entirely reflecting endogenous neural activity. The stimulus employed was a 5 Hz frequency-modulated tone, with brief masker probes (noise bursts) occasionally added. For half the probes, the rhythmic frequency modulation was moreover removed. Listeners reported whether the tone masked by each probe was perceived as being rhythmic or not. Time-frequency analysis of neural responses obtained by magnetoencephalography (MEG) shows that for maskers without the underlying acoustic rhythm, trials where rhythm was nonetheless perceived show higher evoked sustained rhythmic power than trials for which no rhythm was reported. The results support a model in which perceptual filling-in is aided by differential co-modulations of cortical activity at rates directly relevant to human speech communication. We propose that the presence of rhythmically-modulated neural dynamics predicts the subjective experience of a rhythmically modulated sound in real time, even when the perceptual experience is not supported by corresponding sensory data.

A roadmap for the study of conscious audition and its neural basis

How and which aspects of neural activity give rise to subjective perceptual experience-i.e. conscious perception-is a fundamental question of neuroscience. To date, the vast majority of work concerning this question has come from vision, raising the issue of generalizability of prominent resulting theories. However, recent work has begun to shed light on the neural processes subserving conscious perception in other modalities, particularly audition. Here, we outline a roadmap for the future study of conscious auditory perception and its neural basis, paying particular attention to how conscious perception emerges (and of which elements or groups of elements) in complex auditory scenes. We begin by discussing the functional role of the auditory system, particularly as it pertains to conscious perception. Next, we ask: what are the phenomena that need to be explained by a theory of conscious auditory perception? After surveying the available literature for candidate neural correlates, we end by considering the implications that such results have for a general theory of conscious perception as well as prominent outstanding questions and what approaches/techniques can best be used to address them.This article is part of the themed issue 'Auditory and visual scene analysis'.

High-frequency neural activity predicts word parsing in ambiguous speech streams

During speech listening, the brain parses a continuous acoustic stream of information into computational units (e.g., syllables or words) necessary for speech comprehension. Recent neuroscientific hypotheses have proposed that neural oscillations contribute to speech parsing, but whether they do so on the basis of acoustic cues (bottom-up acoustic parsing) or as a function of available linguistic representations (top-down linguistic parsing) is unknown. In this magnetoencephalography study, we contrasted acoustic and linguistic parsing using bistable speech sequences. While listening to the speech sequences, participants were asked to maintain one of the two possible speech percepts through volitional control. We predicted that the tracking of speech dynamics by neural oscillations would not only follow the acoustic properties but also shift in time according to the participant's conscious speech percept. Our results show that the latency of high-frequency activity (specifically, beta and gamma bands) varied as a function of the perceptual report. In contrast, the phase of low-frequency oscillations was not strongly affected by top-down control. Whereas changes in low-frequency neural oscillations were compatible with the encoding of prelexical segmentation cues, high-frequency activity specifically informed on an individual's conscious speech percept.

Tonotopic and Field-Specific Representation of Long-Lasting Sustained Activity in Rat Auditory Cortex

Cortical information processing of the onset, offset, and continuous plateau of an acoustic stimulus should play an important role in acoustic object perception. To date, transient activities responding to the onset and offset of a sound have been well investigated and cortical subfields and topographic representation in these subfields, such as place code of sound frequency, have been well characterized. However, whether these cortical subfields with tonotopic representation are inherited in the sustained activities that follow transient activities and persist during the presentation of a long-lasting stimulus remains unknown, because sustained activities do not exhibit distinct, reproducible, and time-locked responses in their amplitude to be characterized by grand averaging. To address this gap in understanding, we attempted to decode sound information from densely mapped sustained activities in the rat auditory cortex using a sparse parameter estimation method called sparse logistic regression (SLR), and investigated whether and how these activities represent sound information. A microelectrode array with a grid of 10 × 10 recording sites within an area of 4.0 mm × 4.0 mm was implanted in the fourth layer of the auditory cortex in rats under isoflurane anesthesia. Sustained activities in response to long-lasting constant pure tones were recorded. SLR then was applied to discriminate the sound-induced band-specific power or phase-locking value from those of spontaneous activities. The highest decoding performance was achieved in the high-gamma band, indicating that cortical inhibitory interneurons may contribute to the sparse tonotopic representation in sustained activities by mediating synchronous activities. The estimated parameter in the SLR decoding revealed that the informative recording site had a characteristic frequency close to the test frequency. In addition, decoding of the four test frequencies demonstrated that the decoding performance of the SLR deteriorated when the test frequencies were close, supporting the hypothesis that the sustained activities were organized in a tonotopic manner. Finally, unlike transient activities, sustained activities were more informative in the belt than in the core region, indicating that higher-order auditory areas predominate over lower-order areas during sustained activities. Taken together, our results indicate that the auditory cortex processes sound information tonotopically and in a hierarchical manner.

Role of contextual cues on the perception of spectrally reduced interrupted speech

Understanding speech within an auditory scene is constantly challenged by interfering noise in suboptimal listening environments when noise hinders the continuity of the speech stream. In such instances, a typical auditory-cognitive system perceptually integrates available speech information and "fills in" missing information in the light of semantic context. However, individuals with cochlear implants (CIs) find it difficult and effortful to understand interrupted speech compared to their normal hearing counterparts. This inefficiency in perceptual integration of speech could be attributed to further degradations in the spectral-temporal domain imposed by CIs making it difficult to utilize the contextual evidence effectively. To address these issues, 20 normal hearing adults listened to speech that was spectrally reduced and spectrally reduced interrupted in a manner similar to CI processing. The Revised Speech Perception in Noise test, which includes contextually rich and contextually poor sentences, was used to evaluate the influence of semantic context on speech perception. Results indicated that listeners benefited more from semantic context when they listened to spectrally reduced speech alone. For the spectrally reduced interrupted speech, contextual information was not as helpful under significant spectral reductions, but became beneficial as the spectral resolution improved. These results suggest top-down processing facilitates speech perception up to a point, and it fails to facilitate speech understanding when the speech signals are significantly degraded.

Theta oscillations accompanying concurrent auditory stream segregation

The ability to isolate a single sound source among concurrent sources is crucial for veridical auditory perception. The present study investigated the event-related oscillations evoked by complex tones, which could be perceived as a single sound and tonal complexes with cues promoting the perception of two concurrent sounds by inharmonicity, onset asynchrony, and/or perceived source location difference of the components tones. In separate task conditions, participants performed a visual change detection task (visual control), watched a silent movie (passive listening) or reported for each tone whether they perceived one or two concurrent sounds (active listening). In two time windows, the amplitude of theta oscillation was modulated by the presence vs. absence of the cues: 60-350ms/6-8Hz (early) and 350-450ms/4-8Hz (late). The early response appeared both in the passive and the active listening conditions; it did not closely match the task performance; and it had a fronto-central scalp distribution. The late response was only elicited in the active listening condition; it closely matched the task performance; and it had a centro-parietal scalp distribution. The neural processes reflected by these responses are probably involved in the processing of concurrent sound segregation cues, in sound categorization, and response preparation and monitoring. The current results are compatible with the notion that theta oscillations mediate some of the processes involved in concurrent sound segregation.

Female túngara frogs do not experience the continuity illusion

In humans and some nonhuman vertebrates, a sound containing brief silent gaps can be rendered perceptually continuous by inserting noise into the gaps. This so-called "continuity illusion" arises from a phenomenon known as "auditory induction" and results in the perception of complete auditory objects despite fragmentary or incomplete acoustic information. Previous studies of auditory induction in gray treefrogs (Hyla versicolor and H. chrysoscelis) have demonstrated an absence of this phenomenon. These treefrog species produce pulsatile (noncontinuous) vocalizations, whereas studies of auditory induction in other taxa, including humans, often present continuous sounds (e.g., frequency-modulated sweeps). This study investigated the continuity illusion in a frog (Physalaemus pustulosus) with an advertisement vocalization that is naturally continuous and thus similar to the tonal sweeps used in human psychophysical studies of auditory induction. In a series of playback experiments, female subjects were presented with sets of stimuli that included complete calls, calls with silent gaps, and calls with silent gaps filled with noise. The results failed to provide evidence of auditory induction. Current evidence, therefore, suggests that mammals and birds experience auditory induction, but frogs may not. This emerging pattern of taxonomic differences is considered in light of potential methodological, neurophysiological, and functional explanations.

Divided attention disrupts perceptual encoding during speech recognition

Performing a secondary task while listening to speech has a detrimental effect on speech processing, but the locus of the disruption within the speech system is poorly understood. Recent research has shown that cognitive load imposed by a concurrent visual task increases dependency on lexical knowledge during speech processing, but it does not affect lexical activation per se. This suggests that "lexical drift" under cognitive load occurs either as a post-lexical bias at the decisional level or as a secondary consequence of reduced perceptual sensitivity. This study aimed to adjudicate between these alternatives using a forced-choice task that required listeners to identify noise-degraded spoken words with or without the addition of a concurrent visual task. Adding cognitive load increased the likelihood that listeners would select a word acoustically similar to the target even though its frequency was lower than that of the target. Thus, there was no evidence that cognitive load led to a high-frequency response bias. Rather, cognitive load seems to disrupt sublexical encoding, possibly by impairing perceptual acuity at the auditory periphery.

Visual context due to speech-reading suppresses the auditory response to acoustic interruptions in speech

Speech reading enhances auditory perception in noise. One means by which this perceptual facilitation comes about is through information from visual networks reinforcing the encoding of the congruent speech signal by ignoring interfering acoustic signals. We tested this hypothesis neurophysiologically by acquiring EEG while individuals listened to words with a fixed portion of each word replaced by white noise. Congruent (meaningful) or incongruent (reversed frames) mouth movements accompanied the words. Individuals judged whether they heard the words as continuous (illusion) or interrupted (illusion failure) through the noise. We hypothesized that congruent, as opposed to incongruent, mouth movements should further enhance illusory perception by suppressing the auditory cortex's response to interruption onsets and offsets. Indeed, we found that the N1 auditory evoked potential (AEP) to noise onsets and offsets was reduced when individuals experienced the illusion during congruent, but not incongruent, audiovisual streams. This N1 inhibitory effect was most prominent at noise offsets, suggesting that visual influences on auditory perception are instigated to a greater extent during noisy periods. These findings suggest that visual context due to speech-reading disengages (inhibits) neural processes associated with interfering sounds (e.g., noisy interruptions) during speech perception.

No time-stretching illusion when a tone is followed by a noise

A sine tone is perceived as longer when it is preceded by a more intense noise than when presented in isolation. This is the time-stretching illusion. We conducted an experiment where the method of constant stimuli was used to examine whether a tone would also be stretched when it was followed by a noise. The duration of a tone was overestimated when it was preceded by a noise, but not when followed by a noise or when located between two consecutive noises. Moreover, the increasing of the noise intensity (from -6 to +6 dB) relative to the tone intensity resulted in larger overestimations, but only in the condition where a tone was preceded by a noise. In brief, the duration of a tone is stretched when this tone is preceded by a noise and if this tone is not followed by a noise.

Functional imaging of auditory scene analysis

Our auditory system is constantly faced with the task of decomposing the complex mixture of sound arriving at the ears into perceptually independent streams constituting accurate representations of individual sound sources. This decomposition, termed auditory scene analysis, is critical for both survival and communication, and is thought to underlie both speech and music perception. The neural underpinnings of auditory scene analysis have been studied utilizing invasive experiments with animal models as well as non-invasive (MEG, EEG, and fMRI) and invasive (intracranial EEG) studies conducted with human listeners. The present article reviews human neurophysiological research investigating the neural basis of auditory scene analysis, with emphasis on two classical paradigms termed streaming and informational masking. Other paradigms - such as the continuity illusion, mistuned harmonics, and multi-speaker environments - are briefly addressed thereafter. We conclude by discussing the emerging evidence for the role of auditory cortex in remapping incoming acoustic signals into a perceptual representation of auditory streams, which are then available for selective attention and further conscious processing. This article is part of a Special Issue entitled Human Auditory Neuroimaging.

Narrowed Expectancies under Degraded Speech: Revisiting the N400

Under adverse listening conditions, speech comprehension profits from the expectancies that listeners derive from the semantic context. However, the neurocognitive mechanisms of this semantic benefit are unclear: How are expectancies formed from context and adjusted as a sentence unfolds over time under various degrees of acoustic degradation? In an EEG study, we modified auditory signal degradation by applying noise-vocoding (severely degraded: four-band, moderately degraded: eight-band, and clear speech). Orthogonal to that, we manipulated the extent of expectancy: strong or weak semantic context (±con) and context-based typicality of the sentence-last word (high or low: ±typ). This allowed calculation of two distinct effects of expectancy on the N400 component of the evoked potential. The sentence-final N400 effect was taken as an index of the neural effort of automatic word-into-context integration; it varied in peak amplitude and latency with signal degradation and was not reliably observed in response to severely degraded speech. Under clear speech conditions in a strong context, typical and untypical sentence completions seemed to fulfill the neural prediction, as indicated by N400 reductions. In response to moderately degraded signal quality, however, the formed expectancies appeared more specific: Only typical (+con +typ), but not the less typical (+con −typ) context–word combinations led to a decrease in the N400 amplitude. The results show that adverse listening “narrows,” rather than broadens, the expectancies about the perceived speech signal: limiting the perceptual evidence forces the neural system to rely on signal-driven expectancies, rather than more abstract expectancies, while a sentence unfolds over time.

Frequency-change aftereffect produced by adaptation to real and illusory unidirectional frequency sweeps

It was examined whether illusory and real continuities induce the frequency-change aftereffect, in which repeated exposure to a frequency sweep results in a shift in the perceived frequency change direction of a subsequent test sound. The magnitude of the aftereffect for different types of adaptors ("real sweep," "illusory sweep," and "sweep with gap") was compared. Listeners judged the direction of a frequency change of the test sound and showed a significant aftereffect only for the "real sweep" adaptors. The results suggest that the illusory sweeps are processed after the stage of frequency-change detection.

Behavioral evidence for auditory induction in a species of rodent: Mongolian gerbil (Meriones unguiculatus)

When a segment of sound of interest is interrupted by a loud extraneous noise, humans perceive that the missing sound continues during the intrusive noise. This restoration of auditory information occurs in perceptions of both speech and non-speech sounds (e.g., tone bursts), a phenomenon referred to as auditory induction. In this study, Mongolian gerbils were trained with standard Go/No-Go operant conditioning to discriminate continuous tone bursts (the Go stimulus) from tone bursts with a silent gap in the middle (the No-Go stimulus). Noise was added to Go and No-Go stimuli to determine the condition under which induction would occur. The Mongolian gerbils engaged in Go responses to No-Go stimuli only when the noise spectrally surrounding the tone was of the same duration as the silent portion of the No-Go stimulus; these results match those previously reported in primates (humans and macaque monkeys). The result presents not only the evidence of the auditory induction in a rodent species but also suggests that similar mechanisms for restoring missing sounds are shared among mammals. Additionally, our findings demonstrated that the rodent can serve as a valuable animal model for future studies of perceptual restoration.

Hearing an illusory vowel in noise: suppression of auditory cortical activity

Human hearing is constructive. For example, when a voice is partially replaced by an extraneous sound (e.g., on the telephone due to a transmission problem), the auditory system may restore the missing portion so that the voice can be perceived as continuous (Miller and Licklider, 1950; for review, see Bregman, 1990; Warren, 1999). The neural mechanisms underlying this continuity illusion have been studied mostly with schematic stimuli (e.g., simple tones) and are still a matter of debate (for review, see Petkov and Sutter, 2011). The goal of the present study was to elucidate how these mechanisms operate under more natural conditions. Using psychophysics and electroencephalography (EEG), we assessed simultaneously the perceived continuity of a human vowel sound through interrupting noise and the concurrent neural activity. We found that vowel continuity illusions were accompanied by a suppression of the 4 Hz EEG power in auditory cortex (AC) that was evoked by the vowel interruption. This suppression was stronger than the suppression accompanying continuity illusions of a simple tone. Finally, continuity perception and 4 Hz power depended on the intactness of the sound that preceded the vowel (i.e., the auditory context). These findings show that a natural sound may be restored during noise due to the suppression of 4 Hz AC activity evoked early during the noise. This mechanism may attenuate sudden pitch changes, adapt the resistance of the auditory system to extraneous sounds across auditory scenes, and provide a useful model for assisted hearing devices.

β- And γ-band EEG power predicts illusory auditory continuity perception

Because acoustic landscapes are complex and rapidly changing, auditory systems have evolved mechanisms that permit rapid detection of novel sounds, sound source segregation, and perceptual restoration of sounds obscured by noise. Perceptual restoration is particularly important in noisy environments because it allows organisms to track sounds over time even when they are masked. The continuity illusion is a striking example of perceptual restoration with sounds perceived as intact even when parts of them have been replaced by gaps and rendered inaudible by being masked by an extraneous sound. The mechanisms of auditory filling-in are complex and are currently not well-understood. The present study used the high temporal resolution of EEG to examine brain activity related to continuity illusion perception. Masking noise loudness was adjusted individually for each subject so that physically identical sounds on some trials elicited a continuity illusion (failure to detect a gap in a sound) and on other trials resulted in correct gap detection. This design ensured that any measurable differences in brain activity would be due to perceptual differences rather than physical differences among stimuli. We found that baseline activity recorded immediately before presentation of the stimulus significantly predicted the occurrence of the continuity illusion in 10 out of 14 participants based on power differences in γ-band EEG (34-80 Hz). Across all participants, power in the β and γ (12- to 80-Hz range) was informative about the subsequent perceptual decision. These data suggest that a subject's baseline brain state influences the strength of continuity illusions.

Neural restoration of degraded audiovisual speech

When speech is interrupted by noise, listeners often perceptually "fill-in" the degraded signal, giving an illusion of continuity and improving intelligibility. This phenomenon involves a neural process in which the auditory cortex (AC) response to onsets and offsets of acoustic interruptions is suppressed. Since meaningful visual cues behaviorally enhance this illusory filling-in, we hypothesized that during the illusion, lip movements congruent with acoustic speech should elicit a weaker AC response to interruptions relative to static (no movements) or incongruent visual speech. AC response to interruptions was measured as the power and inter-trial phase consistency of the auditory evoked theta band (4-8 Hz) activity of the electroencephalogram (EEG) and the N1 and P2 auditory evoked potentials (AEPs). A reduction in the N1 and P2 amplitudes and in theta phase-consistency reflected the perceptual illusion at the onset and/or offset of interruptions regardless of visual condition. These results suggest that the brain engages filling-in mechanisms throughout the interruption, which repairs degraded speech lasting up to ~250 ms following the onset of the degradation. Behaviorally, participants perceived speech continuity over longer interruptions for congruent compared to incongruent or static audiovisual streams. However, this specific behavioral profile was not mirrored in the neural markers of interest. We conclude that lip-reading enhances illusory perception of degraded speech not by altering the quality of the AC response, but by delaying it during degradations so that longer interruptions can be tolerated.

Effects of inducer continuity on auditory stream segregation: comparison of physical and perceived continuity in different contexts

The factors influencing the stream segregation of discrete tones and the perceived continuity of discrete tones as continuing through an interrupting masker are well understood as separate phenomena. Two experiments tested whether perceived continuity can influence the build-up of stream segregation by manipulating the perception of continuity during an induction sequence and measuring streaming in a subsequent test sequence comprising three triplets of low and high frequency tones (LHL-[ellipsis (horizontal)]). For experiment 1, a 1.2-s standard induction sequence comprising six 100-ms L-tones strongly promoted segregation, whereas a single extended L-inducer (1.1 s plus 100-ms silence) did not. Segregation was similar to that following the single extended inducer when perceived continuity was evoked by inserting noise bursts between the individual tones. Reported segregation increased when the noise level was reduced such that perceived continuity no longer occurred. Experiment 2 presented a 1.3-s continuous inducer created by bridging the 100-ms silence between an extended L-inducer and the first test-sequence tone. This configuration strongly promoted segregation. Segregation was also increased by filling the silence after the extended inducer with noise, such that it was perceived like a bridging inducer. Like physical continuity, perceived continuity can promote or reduce test-sequence streaming, depending on stimulus context.

Perceived tonal continuity through two noise bursts separated by silence

Three experiments measured the perceived continuity of two pure tones "flankers" through a masker containing a silence. Experiment 1 used a 2I-2AFC procedure; one interval contained two noise bursts separated by a silent gap, and the other contained two noise bursts separated by a tone of the same duration as the silence. Discrimination between masker conditions was very accurate when the flankers were absent but was impaired substantially when the flankers were present. This was taken as evidence that illusory flanker continuity during the silent gap was heard as similar to the physical presence of a tone in the gap. In experiment 2, performance remained poor when the flankers were frequency glides aligned along a common trajectory. Performance improved significantly when the flankers were misaligned in trajectory. In experiment 3, listeners rated directly perceived flanker continuity. Strong continuity was reported in the silent gap conditions for which poor performance had been observed in experiments 1 and 2. These findings show that continuity may be heard through a masker that cannot mask a physically continuous tone but can mask the flankers' offset and onset. The results are explained in terms of the perceptual grouping of onsets and offsets of the flankers.

Auditory cortex encodes the perceptual interpretation of ambiguous sound

The confounding of physical stimulus characteristics and perceptual interpretations of stimuli poses a problem for most neuroscientific studies of perception. In the auditory domain, this pertains to the entanglement of acoustics and percept. Traditionally, most study designs have relied on cognitive subtraction logic, which demands the use of one or more comparisons between stimulus types. This does not allow for a differentiation between effects due to acoustic differences (i.e., sensation) and those due to conscious perception. To overcome this problem, we used functional magnetic resonance imaging (fMRI) in humans and pattern-recognition analysis to identify activation patterns that encode the perceptual interpretation of physically identical, ambiguous sounds. We show that it is possible to retrieve the perceptual interpretation of ambiguous phonemes-information that is fully subjective to the listener-from fMRI measurements of brain activity in auditory areas in the superior temporal cortex, most prominently on the posterior bank of the left Heschl's gyrus and sulcus and in the adjoining left planum temporale. These findings suggest that, beyond the basic acoustic analysis of sounds, constructive perceptual processes take place in these relatively early cortical auditory networks. This disagrees with hierarchical models of auditory processing, which generally conceive of these areas as sets of feature detectors, whose task is restricted to the analysis of physical characteristics and the structure of sounds.

Individual differences in sound-in-noise perception are related to the strength of short-latency neural responses to noise

Important sounds can be easily missed or misidentified in the presence of extraneous noise. We describe an auditory illusion in which a continuous ongoing tone becomes inaudible during a brief, non-masking noise burst more than one octave away, which is unexpected given the frequency resolution of human hearing. Participants strongly susceptible to this illusory discontinuity did not perceive illusory auditory continuity (in which a sound subjectively continues during a burst of masking noise) when the noises were short, yet did so at longer noise durations. Participants who were not prone to illusory discontinuity showed robust early electroencephalographic responses at 40-66 ms after noise burst onset, whereas those prone to the illusion lacked these early responses. These data suggest that short-latency neural responses to auditory scene components reflect subsequent individual differences in the parsing of auditory scenes.

Recalibration of the auditory continuity illusion: sensory and decisional effects

An interrupted sound can be perceived as continuous when noise masks the interruption, creating an illusion of continuity. Recent findings have shown that adaptor sounds preceding an ambiguous target sound can influence listeners' rating of target continuity. However, it remains unclear whether these aftereffects on perceived continuity influence sensory processes, decisional processes (i.e., criterion shifts), or both. The present study addressed this question. Results show that the target sound was more likely to be rated as 'continuous' when preceded by adaptors that were perceived as clearly discontinuous than when it was preceded by adaptors that were heard (illusorily or veridically) as continuous. Detection-theory analyses indicated that these contrastive aftereffects reflect a combination of sensory and decisional processes. The contrastive sensory aftereffect persisted even when adaptors and targets were presented to opposite ears, suggesting a neural origin in structures that receive binaural inputs. Finally, physically identical but perceptually ambiguous adaptors that were rated as 'continuous' induced more reports of target continuity than adaptors that were rated as 'discontinuous'. This assimilative aftereffect was purely decisional. These findings confirm that judgments of auditory continuity can be influenced by preceding events, and reveal that these aftereffects have both sensory and decisional components.

Tracking vocal pitch through noise: neural correlates in nonprimary auditory cortex

In natural environments, a sound can be heard as stable despite the presence of other occasionally louder sounds. For example, when a portion in a voice is replaced by masking noise, the interrupted voice may still appear illusorily continuous. Previous research found that continuity illusions of simple interrupted sounds, such as tones, are accompanied by weaker activity in the primary auditory cortex (PAC) during the interruption than veridical discontinuity percepts of these sounds. Here, we studied whether continuity illusions of more natural and more complex sounds also emerge from this mechanism. We used psychophysics and functional magnetic resonance imaging in humans to measure simultaneously continuity ratings and blood oxygenation level-dependent activity to vowels that were partially replaced by masking noise. Consistent with previous results on tone continuity illusions, we found listeners' reports of more salient vowel continuity illusions associated with weaker activity in auditory cortex (compared with reports of veridical discontinuity percepts of physically identical stimuli). In contrast to the reduced activity to tone continuity illusions in PAC, this reduction was localized in the right anterolateral Heschl's gyrus, a region that corresponds more to the non-PAC. Our findings suggest that the ability to hear differently complex sounds as stable during other louder sounds may be attributable to a common suppressive mechanism that operates at different levels of sound representation in auditory cortex.

Evolutionary conservation and neuronal mechanisms of auditory perceptual restoration

Auditory perceptual 'restoration' occurs when the auditory system restores an occluded or masked sound of interest. Behavioral work on auditory restoration in humans began over 50 years ago using it to model a noisy environmental scene with competing sounds. It has become clear that not only humans experience auditory restoration: restoration has been broadly conserved in many species. Behavioral studies in humans and animals provide a necessary foundation to link the insights being obtained from human EEG and fMRI to those from animal neurophysiology. The aggregate of data resulting from multiple approaches across species has begun to clarify the neuronal bases of auditory restoration. Different types of neural responses supporting restoration have been found, supportive of multiple mechanisms working within a species. Yet a general principle has emerged that responses correlated with restoration mimic the response that would have been given to the uninterrupted sound of interest. Using the same technology to study different species will help us to better harness animal models of 'auditory scene analysis' to clarify the conserved neural mechanisms shaping the perceptual organization of sound and to advance strategies to improve hearing in natural environmental settings.

Testing an auditory illusion in frogs: Perceptual restoration or sensory bias?

The human auditory system perceptually restores short deleted segments of speech and other sounds (e.g. tones) when the resulting silent gaps are filled by a potential masking noise. When this phenomenon, known as 'auditory induction', occurs, listeners experience the illusion of hearing an ongoing sound continuing through the interrupting noise even though the perceived sound is not physically present. Such illusions suggest that a key function of the auditory system is to allow listeners to perceive complete auditory objects with incomplete acoustic information, as may often be the case in multisource acoustic environments. At present, however, we know little about the possible functions of auditory induction in the sound-mediated behaviours of animals. The present study used two-choice phonotaxis experiments to test the hypothesis that female grey treefrogs, Hyla chrysoscelis, experience the illusory perceptual restoration of discrete pulses in the male advertisement call when pulses are deleted and replaced by a potential masking noise. While added noise restored some attractiveness to calls with missing pulses, there was little evidence to suggest that the frogs actually experienced the illusion of perceiving the missing pulses. Instead, the added noise appeared to function as an acoustic appendage that made some calls more attractive than others as a result of sensory biases, the expression of which depended on the temporal order and acoustic structure of the added appendages.

Discrimination of speech stimuli based on neuronal response phase patterns depends on acoustics but not comprehension

Speech stimuli give rise to neural activity in the listener that can be observed as waveforms using magnetoencephalography. Although waveforms vary greatly from trial to trial due to activity unrelated to the stimulus, it has been demonstrated that spoken sentences can be discriminated based on theta-band (3-7 Hz) phase patterns in single-trial response waveforms. Furthermore, manipulations of the speech signal envelope and fine structure that reduced intelligibility were found to produce correlated reductions in discrimination performance, suggesting a relationship between theta-band phase patterns and speech comprehension. This study investigates the nature of this relationship, hypothesizing that theta-band phase patterns primarily reflect cortical processing of low-frequency (<40 Hz) modulations present in the acoustic signal and required for intelligibility, rather than processing exclusively related to comprehension (e.g., lexical, syntactic, semantic). Using stimuli that are quite similar to normal spoken sentences in terms of low-frequency modulation characteristics but are unintelligible (i.e., their time-inverted counterparts), we find that discrimination performance based on theta-band phase patterns is equal for both types of stimuli. Consistent with earlier findings, we also observe that whereas theta-band phase patterns differ across stimuli, power patterns do not. We use a simulation model of the single-trial response to spoken sentence stimuli to demonstrate that phase-locked responses to low-frequency modulations of the acoustic signal can account not only for the phase but also for the power results. The simulation offers insight into the interpretation of the empirical results with respect to phase-resetting and power-enhancement models of the evoked response.