Seeing speech: visual information from lip movements modifies activity in the human auditory cortex

Temporal window of integration in auditory-visual speech perception

Forty-three normal hearing participants were tested in two experiments, which focused on temporal coincidence in auditory visual (AV) speech perception. In these experiments, audio recordings of/pa/and/ba/were dubbed onto video recordings of /ba/or/ga/, respectively (ApVk, AbVg), to produce the illusory "fusion" percepts /ta/, or /da/ [McGurk, H., & McDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-747]. In Experiment 1, an identification task using McGurk pairs with asynchronies ranging from -467 ms (auditory lead) to +467 ms was conducted. Fusion responses were prevalent over temporal asynchronies from -30 ms to +170 ms and more robust for audio lags. In Experiment 2, simultaneity judgments for incongruent and congruent audiovisual tokens (AdVd, AtVt) were collected. McGurk pairs were more readily judged as asynchronous than congruent pairs. Characteristics of the temporal window over which simultaneity and fusion responses were maximal were quite similar, suggesting the existence of a 200 ms duration asymmetric bimodal temporal integration window.

Die audiovisuelle Sprachverarbeitung bei Patienten mit Cochlea-Implantat

Zusammenfassung: Mit Hilfe der elektrischen Reizung des Hörnervs durch implantierte Elektroden in die Hörschnecke (Cochlea-Implantat: CI) kann bei Ertaubten die Fähigkeit zur Wahrnehmung akustischer Sprachlaute wieder hergestellt werden. Aufgrund der herabgesetzten akustische Qualität der Signale ziehen diese zusätzliche visuelle Informationen heran. Akustische Sprachreize (zweisilbige Substantive) wurden zeitgleich zu einem Video-Segment mit dem Gesicht des Sprechers dargeboten, das entweder dem akustischen Wort kongruente (z. B. Audio: Hotel, Video: Hotel) oder inkongruente Information (z. B. Audio: Hotel, Video: Insel) aussprach. Die Analyse der Verhaltensdaten ergab, dass CI-Patienten deutlich von der zusätzlichen Darbietung des Sprechergesichtes profitieren, um Sprachlaute zu verstehen. Auch Normalhörende nutzen visuelle Informationen, vor allem, wenn die akustischen Signale verrauscht und schwer verständlich sind. Die audiovisuelle Sprachverarbeitung löst bei CI-Nutzern und Normalhörenden unterschiedliche Amplitudenverläufe im ereigniskorrelierten Potenzial aus. Dabei zeigen sich vor allem Unterschiede im okzipitalen Bereich, was als Reorganisation nach Hördeprivation bei CI-Patienten verstanden werden kann.

Nonmodularity of the Central Auditory Nervous System

This response to A. T. Cacace and D. J. McFarland (2005) identifies points of agreement and disagreement regarding the concept of modularity in the diagnosis of (central) auditory processing disorder [(C)APD]. We concur that the evaluation of (C)APD must take into consideration the influence of higher order global or pansensory issues on performance on tests of central auditory function. To accomplish this goal, multidisciplinary (e.g., multimodal) testing is an integral part of differential diagnosis of (C)APD. We also agree that the efficiency of diagnostic tests of (C)APD should not be evaluated by imprecise criteria [e.g., "presumed" or "suspected" (C)APD], which do not provide accurate measures of the true sensitivity and specificity of these tests. Our conceptualization and recommendations for clinical practice in this area diverge, however, from that of Cacace and McFarland in a number of pivotal ways. Based on the current limitations of multimodal assessment relative to issues related to scope of practice and test efficiency, as well as the accumulated basic science and clinical literature that demonstrates the nonmodularity and interactive organization of the brain, we recommend use of the sensitized test battery of the central auditory nervous system (CANS) in combination with multidisciplinary testing to differentially diagnose (C)APD and to guide treatment of the disorder. We assert that sensitivity and specificity measures derived from individuals with well-circumscribed lesions of the CANS provide an important guide to establishing the validity of central auditory diagnostic tests. We note that researchers in the area of auditory science and (C)APD must acknowledge the challenges of the clinical arena, and we encourage their continued help to develop diagnostic tools that are both efficient and practical for the differential diagnosis of (C)APD. We conclude that our approach, which combines multidisciplinary evaluation and specific tests of central auditory function that have demonstrated sensitivity and specificity for disorders of the CANS, allows us to identify (and thus rehabilitate) the auditory deficits present in individuals with (C)APD in its "purest" form. It also permits the identification and rehabilitation of auditory deficits in individuals who exhibit auditory perceptual problems that coexist with other processing problems, while ruling out those who perform poorly on auditory tests because of a global, supramodal problem involving cognition, attention, language, memory, or related skills.

Seeing speech affects acoustic information processing in the human brainstem

Afferent auditory processing in the human brainstem is often assumed to be determined by acoustic stimulus features alone and immune to stimulation by other senses or cognitive factors. In contrast, we show that lipreading during speech perception influences early acoustic processing. Event-related brainstem potentials were recorded from ten healthy adults to concordant (acoustic-visual match), conflicting (acoustic-visual mismatch) and unimodal stimuli. Audiovisual (AV) interactions occurred as early as approximately 11 ms post-acoustic stimulation and persisted for the first 30 ms of the response. Furthermore, the magnitude of interaction depended on AV pairings. These findings indicate considerable plasticity in early auditory processing.

Is the auditory sensory memory sensitive to visual information?

The mismatch negativity (MMN) component of auditory event-related brain potentials can be used as a probe to study the representation of sounds in auditory sensory memory (ASM). Yet it has been shown that an auditory MMN can also be elicited by an illusory auditory deviance induced by visual changes. This suggests that some visual information may be encoded in ASM and is accessible to the auditory MMN process. It is not known, however, whether visual information affects ASM representation for any audiovisual event or whether this phenomenon is limited to specific domains in which strong audiovisual illusions occur. To highlight this issue, we have compared the topographies of MMNs elicited by non-speech audiovisual stimuli deviating from audiovisual standards on the visual, the auditory, or both dimensions. Contrary to what occurs with audiovisual illusions, each unimodal deviant elicited sensory-specific MMNs, and the MMN to audiovisual deviants included both sensory components. The visual MMN was, however, different from a genuine visual MMN obtained in a visual-only control oddball paradigm, suggesting that auditory and visual information interacts before the MMN process occurs. Furthermore, the MMN to audiovisual deviants was significantly different from the sum of the two sensory-specific MMNs, showing that the processes of visual and auditory change detection are not completely independent.

Is the integration of heard and seen speech mandatory for infants?

For adults and children, speech perception can be significantly influenced by watching a speaker's mouth movements. While recent reports suggest that infants may be able to integrate heard and seen speech, the current research demonstrates that integration is neither as strong or consistent in infants as it is in adults. Three habituation experiments were conducted. In Experiment 1, female (but not male) infants showed evidence of an adult pattern of integration following habituation with an audiovisual /bi/ and testing with audio /bi/-visual /vi/ (perceived as /vi/ by adults). The interpretation of integration was supported, but only in part, by Experiment 2. In Experiment 3, infants were habituated to a mismatched audio /bi/-visual /vi/ combination and tested on concordant /bi/-/bi/ versus /vi/-/vi/ displays. Here, only male infants showed evidence of integration. These results suggest that an initial mechanism supports integration, but that integration is not mandatory for young infants.

Early modulation of visual cortex by sound: an MEG study

Sound can alter visual perception. This has been recently demonstrated by a strong illusion in which a single flash is perceived as multiple flashes when accompanied by multiple brief sounds. While psychophysical findings on this sound-induced flash illusion indicate that the modulations of visual percept by sound occur at a perceptual processing level, it remains unclear at what level of perceptual processing these interactions occur and what mechanisms mediate them. Here we investigated these questions using MEG. We found modulation of activity in occipital and parietal scalp locations, when comparing illusion trials with visual-alone and auditory-alone trials. This modulation occurred as early as 35-65 ms from the onset of the visual stimulus. Activity was also modulated in the occipital and parietal areas as well as anterior areas at a later ( approximately 150 ms post-stimulus) onset. No significant interactions were observed in occipital and parietal areas in trials in which illusion was not perceived. These results indicate that the auditory alteration of visual perception as reflected by the illusion is associated with modulation of activity in visual cortex. The early onset of these modulations suggests that a feed-forward or lateral circuitry is at least partially involved in these interactions.

Listening to talking faces: motor cortical activation during speech perception

Neurophysiological research suggests that understanding the actions of others harnesses neural circuits that would be used to produce those actions directly. We used fMRI to examine brain areas active during language comprehension in which the speaker was seen and heard while talking (audiovisual) or heard but not seen (audio-alone) or when the speaker was seen talking with the audio track removed (video-alone). We found that audiovisual speech perception activated a network of brain regions that included cortical motor areas involved in planning and executing speech production and areas subserving proprioception related to speech production. These regions included the posterior part of the superior temporal gyrus and sulcus, the pars opercularis, premotor cortex, adjacent primary motor cortex, somatosensory cortex, and the cerebellum. Activity in premotor cortex and posterior superior temporal gyrus and sulcus was modulated by the amount of visually distinguishable phonemes in the stories. None of these regions was activated to the same extent in the audio- or video-alone conditions. These results suggest that integrating observed facial movements into the speech perception process involves a network of multimodal brain regions associated with speech production and that these areas contribute less to speech perception when only auditory signals are present. This distributed network could participate in recognition processing by interpreting visual information about mouth movements as phonetic information based on motor commands that could have generated those movements.

Primary auditory cortex activation by visual speech: an fMRI study at 3???T

Recent studies have yielded contradictory evidence on whether visual speech perception (watching articulatory gestures) can activate the human primary auditory cortex. To circumvent confounds due to inter-individual anatomical variation, we defined our subjects' Heschl's gyri and assessed blood oxygenation-dependent signal changes at 3 T within this confined region during visual speech perception and observation of moving circles. Visual speech perception activated Heschl's gyri in nine subjects, with activation in seven of them extending to the area of primary auditory cortex. Activation was significantly stronger during visual speech perception than during observation of the moving circles. Further, a significant hemisphere by stimulus interaction occurred, suggesting left Heschl's gyrus specialization for visual speech processing.

Visual speech speeds up the neural processing of auditory speech

Synchronous presentation of stimuli to the auditory and visual systems can modify the formation of a percept in either modality. For example, perception of auditory speech is improved when the speaker's facial articulatory movements are visible. Neural convergence onto multisensory sites exhibiting supra-additivity has been proposed as the principal mechanism for integration. Recent findings, however, have suggested that putative sensory-specific cortices are responsive to inputs presented through a different modality. Consequently, when and where audiovisual representations emerge remain unsettled. In combined psychophysical and electroencephalography experiments we show that visual speech speeds up the cortical processing of auditory signals early (within 100 ms of signal onset). The auditory-visual interaction is reflected as an articulator-specific temporal facilitation (as well as a nonspecific amplitude reduction). The latency facilitation systematically depends on the degree to which the visual signal predicts possible auditory targets. The observed auditory-visual data support the view that there exist abstract internal representations that constrain the analysis of subsequent speech inputs. This is evidence for the existence of an "analysis-by-synthesis" mechanism in auditory-visual speech perception.

Modulations of 'late' event-related brain potentials in humans by dynamic audiovisual speech stimuli

Lipreading reliably improve speech perception during face-to-face conversation. Within the range of good dubbing, however, adults tolerate some audiovisual (AV) discrepancies and lipreading, then, can give rise to confusion. We used event-related brain potentials (ERPs) to study the perceptual strategies governing the intermodal processing of dynamic and bimodal speech stimuli, either congruently dubbed or not. Electrophysiological analyses revealed that non-coherent audiovisual dubbings modulated in amplitude an endogenous ERP component, the N300, we compared to a 'N400-like effect' reflecting the difficulty to integrate these conflicting pieces of information. This result adds further support for the existence of a cerebral system underlying 'integrative processes' lato sensu. Further studies should take advantage of this 'N400-like effect' with AV speech stimuli to open new perspectives in the domain of psycholinguistics.

Seeing and hearing others and oneself talk

We studied the modification of auditory perception in three different conditions in twenty subjects. Observing other person's discordant articulatory gestures deteriorated identification of acoustic speech stimuli and modified the auditory percept, causing a strong McGurk effect. A similar effect was found when the subjects watched their own silent articulation in a mirror and acoustic stimuli were simultaneously presented to their ears. Interestingly, a smaller but significant effect was even obtained when the subjects just silently articulated the syllables without visual feedback. On the other hand, observing other person's or one's own concordant articulation and silently articulating a concordant syllable improved identification of the acoustic stimuli. The modification of auditory percepts caused by visual observation of speech and silently articulating it are both suggested to be due to the alteration of activity in the auditory cortex. Our findings support the idea of a close relationship between speech perception and production.

Bimodal speech: early suppressive visual effects in human auditory cortex

While everyone has experienced that seeing lip movements may improve speech perception, little is known about the neural mechanisms by which audiovisual speech information is combined. Event-related potentials (ERPs) were recorded while subjects performed an auditory recognition task among four different natural syllables randomly presented in the auditory (A), visual (V) or congruent bimodal (AV) condition. We found that: (i) bimodal syllables were identified more rapidly than auditory alone stimuli; (ii) this behavioural facilitation was associated with cross-modal [AV-(A+V)] ERP effects around 120-190 ms latency, expressed mainly as a decrease of unimodal N1 generator activities in the auditory cortex. This finding provides evidence for suppressive, speech-specific audiovisual integration mechanisms, which are likely to be related to the dominance of the auditory modality for speech perception. Furthermore, the latency of the effect indicates that integration operates at pre-representational stages of stimulus analysis, probably via feedback projections from visual and/or polymodal areas.

Roughness perception in sounds: behavioral and ERP evidence

The mismatch negativity (MMN) correlates of the perception of roughness, the unpleasant character of sounds caused by the perception of amplitude fluctuation in the range of 20-200 Hz, were studied on the basis of a variation in the degree of modulation (=modulation index m), which is a main parameter influencing roughness. A psychophysical study showed that perceived roughness of tones increased with modulation index for m-values from 0 up to 1.2. For larger values of m, roughness perception remained stable. In a subsequent ERP-study, infrequent amplitude modulated (AM) tones with varying modulation index were presented in the context of a series of pure tones in an ignore condition. The amplitude of the mismatch negativity correlated highly with the roughness ratings (r = -0.93) and did not increase monotonously with increasing modulation index. We conclude that perceived roughness rather than its physical correlate in sounds is reflected by the MMN and that roughness is thus preattentively encoded.

Common Neural Basis for Phoneme Processing in Infants and Adults

Investigating the degree of similarity between infants' and adults' representation of speech is critical to our understanding of infants' ability to acquire language. Phoneme perception plays a crucial role in language processing, and numerous behavioral studies have demonstrated similar capacities in infants and adults, but are these subserved by the same neural substrates or networks? In this article, we review event-related potential (ERP) results obtained in infants during phoneme discrimination tasks and compare them to results from the adult literature. The striking similarities observed both in behavior and ERPs between initial and mature stages suggest a continuity in processing and neural structure. We argue that infants have access at the beginning of life to phonemic representations, which are modified without training or implicit instruction, but by the statistical distributions of speech input in order to converge to the native phonemic categories.

Hearing lips: gamma-band activity during audiovisual speech perception

Auditory pattern changes have been shown to elicit increases in magnetoencephalographic gamma-band activity (GBA) over left inferior frontal cortex, forming part of the putative auditory ventral "what" processing stream. The present study employed a McGurk-type paradigm to assess whether GBA would be associated with subjectively perceived changes even when auditory stimuli remain unchanged. Magnetoencephalograms were recorded in 16 human subjects during audiovisual mismatch perception. Both infrequent visual (auditory /ta/ + visual /pa/) and acoustic deviants (auditory/pa/ + visual /ta/) were compared with frequent audiovisual standards (auditory /ta/ and visual /ta/). Statistical probability mapping revealed spectral amplitude increases at approximately 75 and approximately 78 Hz to visual deviants. GBA to visual deviants peaked 160 ms after auditory stimulus onset over posterior parietal cortex, at 270 ms over occipital areas and at 320 ms over left inferior frontal cortex. The latter GBA enhancement was consistent with the increase observed previously to pure acoustic mismatch, supporting a role of left inferior frontal cortex for the representation of perceived auditory pattern change. The preceding gamma-band changes over posterior areas may reflect processing of incongruent lip movements in visual motion areas and back-projections to earlier visual cortex.

Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants

OBJECTIVE

The existence of a Mismatch Negativity (MMN) evoked by McGurk percepts elicited by audiovisual syllables with constant auditory components has been previously demonstrated with voiced consonants [Clin. Neurophysiol. 113 (2002) 495]. The present study aimed at generalizing such results with voiceless consonants. In a first experiment, the MMN was computed using the classical subtraction method (standard minus deviant). Since results showed a possible contamination by exogenous visual components, a technique preventing from including those components in the differential waveform was used in a second experiment (deviant in sequence minus deviant presented alone).

METHODS

Cortical evoked potentials were recorded using the oddball paradigm on eight adults in three experimental conditions (auditory alone, visual alone and audiovisual) for experiment one and in two conditions (visual alone and audiovisual) for experiment two. Obtaining illusory percepts was confirmed in additional psychophysical experiments.

RESULTS

Significant MMNs were recorded in the three conditions of experiment one, whereas only the audiovisual condition of experiment two gave rise to a significant MMN.

CONCLUSIONS

Provided that the MMN is computed with deviant stimuli only, the present results confirm the elicitation of genuine audiovisual MMN. Possible refractoriness effects and N2b confound have, however, to be controlled for in further studies.

From symbols to sounds: Visual symbolic information activates sound representations

Expectations on forthcoming sounds can speed up responding to environmental changes and can, thus, be a basis for successful adaptation. The present study investigated event-related brain potential (ERP) effects in situations where particular sounds were predicted on the basis of preceding visual information. Subjects had to map scorelike visual symbols to corresponding sounds. The sounds could be either congruent or occasionally incongruent with the corresponding symbol. When the auditory stimulation was incongruent with the visual information, a brain response was elicited starting as early as about 100 ms from the onset of the auditory stimulus. It had a bilateral frontal distribution and a polarity inversion at the mastoids compatible with the assumption of sources in auditory cortex. These results suggest that the auditory system can establish a representation of an expected stimulus on the basis of visual symbolic information.

Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study

During social interaction speech is perceived simultaneously by audition and vision. We studied interactions in the processing of auditory (A) and visual (V) speech signals in the human brain by comparing neuromagnetic responses to phonetically congruent audiovisual (AV) syllables with the arithmetic sum of responses to A and V syllables. Differences between AV and A+V responses were found bilaterally in the auditory cortices 150-200 ms and in the right superior temporal sulcus (STS) 250-600 ms after stimulus onset, showing that both sensory-specific and multisensory regions of the human temporal cortices are involved in AV speech processing. Importantly, our results suggest that AV interaction in the auditory cortex precedes that in the multisensory STS region.

Integration of Letters and Speech Sounds in the Human Brain

Most people acquire literacy skills with remarkable ease, even though the human brain is not evolutionarily adapted to this relatively new cultural phenomenon. Associations between letters and speech sounds form the basis of reading in alphabetic scripts. We investigated the functional neuroanatomy of the integration of letters and speech sounds using functional magnetic resonance imaging (fMRI). Letters and speech sounds were presented unimodally and bimodally in congruent or incongruent combinations. Analysis of single-subject data and group data aligned on the basis of individual cortical anatomy revealed that letters and speech sounds are integrated in heteromodal superior temporal cortex. Interestingly, responses to speech sounds in a modality-specific region of the early auditory cortex were modified by simultaneously presented letters. These results suggest that efficient processing of culturally defined associations between letters and speech sounds relies on neural mechanisms similar to those naturally evolved for integrating audiovisual speech.

Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information

Perception of speech is improved when presentation of the audio signal is accompanied by concordant visual speech gesture information. This enhancement is most prevalent when the audio signal is degraded. One potential means by which the brain affords perceptual enhancement is thought to be through the integration of concordant information from multiple sensory channels in a common site of convergence, multisensory integration (MSI) sites. Some studies have identified potential sites in the superior temporal gyrus/sulcus (STG/S) that are responsive to multisensory information from the auditory speech signal and visual speech movement. One limitation of these studies is that they do not control for activity resulting from attentional modulation cued by such things as visual information signaling the onsets and offsets of the acoustic speech signal, as well as activity resulting from MSI of properties of the auditory speech signal with aspects of gross visual motion that are not specific to place of articulation information. This fMRI experiment uses spatial wavelet bandpass filtered Japanese sentences presented with background multispeaker audio noise to discern brain activity reflecting MSI induced by auditory and visual correspondence of place of articulation information that controls for activity resulting from the above-mentioned factors. The experiment consists of a low-frequency (LF) filtered condition containing gross visual motion of the lips, jaw, and head without specific place of articulation information, a midfrequency (MF) filtered condition containing place of articulation information, and an unfiltered (UF) condition. Sites of MSI selectively induced by auditory and visual correspondence of place of articulation information were determined by the presence of activity for both the MF and UF conditions relative to the LF condition. Based on these criteria, sites of MSI were found predominantly in the left middle temporal gyrus (MTG), and the left STG/S (including the auditory cortex). By controlling for additional factors that could also induce greater activity resulting from visual motion information, this study identifies potential MSI sites that we believe are involved with improved speech perception intelligibility.

Neural processes underlying perceptual enhancement by visual speech gestures

This fMRI study explores brain regions involved with perceptual enhancement afforded by observation of visual speech gesture information. Subjects passively identified words presented in the following conditions: audio-only, audiovisual, audio-only with noise, audiovisual with noise, and visual only. The brain may use concordant audio and visual information to enhance perception by integrating the information in a converging multisensory site. Consistent with response properties of multisensory integration sites, enhanced activity in middle and superior temporal gyrus/sulcus was greatest when concordant audiovisual stimuli were presented with acoustic noise. Activity found in brain regions involved with planning and execution of speech production in response to visual speech presented with degraded or absent auditory stimulation, is consistent with the use of an additional pathway through which speech perception is facilitated by a process of internally simulating the intended speech act of the observed speaker.

Interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans: a magnetoencephalographic study

We investigated the interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans, using magnetoencephalography. We compared the difference in the main component, M100 generated in the auditory cortex, in terms of peak latency, amplitude, dipole location and moment, following the vowel sound_/a/_between two conditions: (1) showing a face with closed mouth; and (2) showing the same face with mouth movement appearing to pronounce/a/using an apparent motion method. We found no significant difference in the M100 component between the two conditions within or between the right and left hemispheres. These findings indicated that the vowel sound perception in the auditory cortex, at least in the primary processing stage, was not affected by viewing mouth movement.

Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity

The ventriloquist illusion arises when sounds are mislocated towards a synchronous but spatially discrepant visual event. Here, we investigated the ventriloquist illusion at a neurophysiological level. The question was whether an illusory shift in sound location was reflected in the auditory mismatch negativity (MMN). An 'oddball' paradigm was used whereby simultaneously presented sounds and flashes coming from the same location served as standard. The deviant consisted of a sound originating from the same source as the standard together with a flash at 20 degrees spatial separation, which evoked an illusory sound shift. This illusory sound shift evoked an MMN closely resembling the MMN evoked by an actual sound shift. A visual-only control condition ruled out that the illusory-evoked MMN was confounded by the visual part of the audiovisual deviant. These results indicate that the crossmodal interaction on which the ventriloquist illusion is based takes place automatically at an early processing stage, within 200 ms after stimulus onset.

Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception

We studied the interactions in neural processing of auditory and visual speech by recording event-related brain potentials (ERPs). Unisensory (auditory - A and visual - V) and audiovisual (AV) vowels were presented to 11 subjects. AV vowels were phonetically either congruent (e.g., acoustic /a/ and visual /a/) or incongruent (e.g., acoustic /a/ and visual /y/). ERPs to AV stimuli and the sum of the ERPs to A and V stimuli (A+V) were compared. Similar ERPs to AV and A+V were hypothesized to indicate independent processing of A and V stimuli. Differences on the other hand would suggest AV interactions. Three deflections, the first peaking at about 85 ms after the A stimulus onset, were significantly larger in the ERPs to A+V than in the ERPs to both congruent and incongruent AV stimuli. We suggest that these differences reflect AV interactions in the processing of general, non-phonetic, features shared by the acoustic and visual stimulus (spatial location, coincidence in time). The first difference in the ERPs to incongruent and congruent AV vowels peaked at 155 ms from the A stimuli onset. This and two later differences are suggested to reflect interactions at phonetic level. The early general AV interactions probably reflect modified activity in the sensory-specific cortices, whereas the later phonetic AV interactions are likely generated in the heteromodal cortices. Thus, our results suggest that sensory-specific and heteromodal brain regions participate in AV speech integration at separate latencies and are sensitive to different features of A and V speech stimuli.

Governing coordination: behavioural principles and neural correlates

The coordination of movement is governed by a coalition of constraints. The expression of these constraints ranges from the concrete--the restricted range of motion offered by the mechanical configuration of our muscles and joints; to the abstract--the difficulty that we experience in combining simple movements into complex rhythms. We seek to illustrate that the various constraints on coordination are complementary and inclusive, and the means by which their expression and interaction are mediated systematically by the integrative action of the central nervous system (CNS). Beyond identifying the general principles at the behavioural level that govern the mutual interplay of constraints, we attempt to demonstrate that these principles have as their foundation specific functional properties of the cortical motor systems. We propose that regions of the brain upstream of the motor cortex may play a significant role in mediating interactions between the functional representations of muscles engaged in sensorimotor coordination tasks. We also argue that activity in these "supramotor" regions may mediate the stabilising role of augmented sensory feedback.

Auditory-visual speech perception examined by fMRI and PET

Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.

Anisotropic functional connections between the auditory cortex and area 18a in rat cerebral slices

We developed a new method to visualize the myeloarchitecture in fresh slices, and investigated the properties of the functional neural connections around the boundary between the primary auditory cortex (area 41) and area 18a in rat cerebral slices. A fresh slice illuminated by near-vertical light was observed with a CCD camera. The translucent images of the slice showed contrast patterns very similar to myeloarchitecture. The boundary between these areas was identified by the well-developed layer IV/V in area 41 but not in area 18a. Antidromic/presynaptic components of the field potentials stimulated and recorded across the areal boundary showed symmetric distribution, while the postsynaptic field potentials in the direction from area 41 to 18a were more prominent than those in the opposite direction in layer II/III. In contrast, the dominant direction of propagation of postsynaptic potentials was from area 18a to 41 in layer V. In the presence of 1 microM bicuculline, an inhibitor of GABA(A) receptors, the polysynaptic activities propagating from area 18a into 41 via layer V were elicited by stimulation of area 18a. The propagation measured by Ca(2+) imaging or field potential recordings was potentiated after both areas 18a and 41 were alternately stimulated several times.

Cortical operational synchrony during audio-visual speech integration

Information from different sensory modalities is processed in different cortical regions. However, our daily perception is based on the overall impression resulting from the integration of information from multiple sensory modalities. At present it is not known how the human brain integrates information from different modalities into a unified percept. Using a robust phenomenon known as the McGurk effect it was shown in the present study that audio-visual synthesis takes place within a distributed and dynamic cortical networks with emergent properties. Various cortical sites within these networks interact with each other by means of so-called operational synchrony (Kaplan, Fingelkurts, Fingelkurts, & Darkhovsky, 1997). The temporal synchronization of cortical operations processing unimodal stimuli at different cortical sites reveals the importance of the temporal features of auditory and visual stimuli for audio-visual speech integration.

Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices

One of the principal functions of the nervous system is to synthesize information from multiple sensory channels into a coherent behavioral and perceptual gestalt. A critical feature of this multisensory synthesis is the sorting and coupling of information derived from the same event. One of the singular features of stimuli conveying such information is their contextual or semantic congruence. Illustrating this fact, subjects are typically faster and more accurate when performing tasks that include congruent compared to incongruent cross-modal stimuli. Using functional magnetic resonance imaging, we demonstrate that activity in select brain areas is sensitive to the contextual congruence among cross-modal cues and to task difficulty. The anterior cingulate gyrus and adjacent medial prefrontal cortices showed significantly greater activity when visual and auditory stimuli were contextually congruent (i.e., matching) than when they were nonmatching. Although activity in these regions was also dependent on task difficulty, showing decreased activity with decreasing task difficulty, the activity changes associated with stimulus congruence predominated.

Electrophysiology and brain imaging of biological motion

The movements of the faces and bodies of other conspecifics provide stimuli of considerable interest to the social primate. Studies of single cells, field potential recordings and functional neuroimaging data indicate that specialized visual mechanisms exist in the superior temporal sulcus (STS) of both human and non-human primates that produce selective neural responses to moving natural images of faces and bodies. STS mechanisms also process simplified displays of biological motion involving point lights marking the limb articulations of animate bodies and geometrical shapes whose motion simulates purposeful behaviour. Facial movements such as deviations in eye gaze, important for gauging an individual's social attention, and mouth movements, indicative of potential utterances, generate particularly robust neural responses that differentiate between movement types. Collectively such visual processing can enable the decoding of complex social signals and through its outputs to limbic, frontal and parietal systems the STS may play a part in enabling appropriate affective responses and social behaviour.

Audio-visual integration in schizophrenia

Integration of information provided simultaneously by audition and vision was studied in a group of 18 schizophrenic patients. They were compared to a control group, consisting of 12 normal adults of comparable age and education. By administering two tasks, each focusing on one aspect of audio-visual integration, the study could differentiate between a spatial integration deficit and a speech-based integration deficit. Experiment 1 studied audio-visual interactions in the spatial localisation of sounds. Experiment 2 investigated integration of auditory and visual speech. The schizophrenic group performed as the control group on the sound localisation task, but in the audio-visual speech task, there was an impairment in lipreading as well as a smaller impact of lipreading on auditory speech information. Combined with findings about functional and neuro-anatomical specificity of intersensory integration, the data suggest that there is an integration deficit in the schizophrenic group that is related to the processing of phonetic information.

Neural substrates of perceptual enhancement by cross-modal spatial attention

Orienting attention involuntarily to the location of a sudden sound improves perception of subsequent visual stimuli that appear nearby. The neural substrates of this cross-modal attention effect were investigated by recording event-related potentials to the visual stimuli using a dense electrode array and localizing their brain sources through inverse dipole modeling. A spatially nonpredictive auditory precue modulated visual-evoked neural activity first in the superior temporal cortex at 120-140 msec and then in the ventral occipital cortex of the fusiform gyrus 15-25 msec later. This spatio-temporal sequence of brain activity suggests that enhanced visual perception produced by the cross-modal orienting of spatial attention results from neural feedback from the multimodal superior temporal cortex to the visual cortex of the ventral processing stream.

Reading speech from still and moving faces: the neural substrates of visible speech

Speech is perceived both by ear and by eye. Unlike heard speech, some seen speech gestures can be captured in stilled image sequences. Previous studies have shown that in hearing people, natural time-varying silent seen speech can access the auditory cortex (left superior temporal regions). Using functional magnetic resonance imaging (fMRI), the present study explored the extent to which this circuitry was activated when seen speech was deprived of its time-varying characteristics. In the scanner, hearing participants were instructed to look for a prespecified visible speech target sequence ("voo" or "ahv") among other monosyllables. In one condition, the image sequence comprised a series of stilled key frames showing apical gestures (e.g., separate frames for "v" and "oo" [from the target] or "ee" and "m" [i.e., from nontarget syllables]). In the other condition, natural speech movement of the same overall segment duration was seen. In contrast to a baseline condition in which the letter "V" was superimposed on a resting face, stilled speech face images generated activation in posterior cortical regions associated with the perception of biological movement, despite the lack of apparent movement in the speech image sequence. Activation was also detected in traditional speech-processing regions including the left inferior frontal (Broca's) area, left superior temporal sulcus (STS), and left supramarginal gyrus (the dorsal aspect of Wernicke's area). Stilled speech sequences also generated activation in the ventral premotor cortex and anterior inferior parietal sulcus bilaterally. Moving faces generated significantly greater cortical activation than stilled face sequences, and in similar regions. However, a number of differences between stilled and moving speech were also observed. In the visual cortex, stilled faces generated relatively more activation in primary visual regions (V1/V2), while visual movement areas (V5/MT+) were activated to a greater extent by moving faces. Cortical regions activated more by naturally moving speaking faces included the auditory cortex (Brodmann's Areas 41/42; lateral parts of Heschl's gyrus) and the left STS and inferior frontal gyrus. Seen speech with normal time-varying characteristics appears to have preferential access to "purely" auditory processing regions specialized for language, possibly via acquired dynamic audiovisual integration mechanisms in STS. When seen speech lacks natural time-varying characteristics, access to speech-processing systems in the left temporal lobe may be achieved predominantly via action-based speech representations, realized in the ventral premotor cortex.

The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects

Early visual cortex can be recruited by meaningful sounds in the absence of visual information. This occurs in particular in cochlear implant (CI) patients whose dependency on visual cues in speech comprehension is increased. Such cross-modal interaction mirrors the response of early auditory cortex to mouth movements (speech reading) and may reflect the natural expectancy of the visual counterpart of sounds, lip movements. Here we pursue the hypothesis that visual activations occur specifically in response to meaningful sounds. We performed PET in both CI patients and controls, while subjects listened either to their native language or to a completely unknown language. A recruitment of early visual cortex, the left posterior inferior temporal gyrus (ITG) and the left superior parietal cortex was observed in both groups. While no further activation occurred in the group of normal-hearing subjects, CI patients additionally recruited the right perirhinal/fusiform and mid-fusiform, the right temporo-occipito-parietal (TOP) junction and the left inferior prefrontal cortex (LIPF, Broca's area). This study confirms a participation of visual cortical areas in semantic processing of speech sounds. Observation of early visual activation in normal-hearing subjects shows that auditory-to-visual cross-modal effects can also be recruited under natural hearing conditions. In cochlear implant patients, speech activates the mid-fusiform gyrus in the vicinity of the so-called face area. This suggests that specific cross-modal interaction involving advanced stages in the visual processing hierarchy develops after cochlear implantation and may be the correlate of increased usage of lip-reading.

On the neuronal basis for multisensory convergence: a brief overview

For multisensory stimulation to effect perceptual and behavioral responses, information from the different sensory systems must converge on individual neurons. A great deal is already known regarding processing within the separate sensory systems, as well as about many of the integrative and perceptual/behavioral effects of multisensory processing. However, virtually nothing is known about the functional architecture that underlies multisensory convergence even though it is an integral step to this processing sequence. This paper seeks to summarize the findings pertinent to multisensory convergence, and to initiate the identification of specific convergence patterns that may underlie different multisensory perceptual and behavioral effects.

Facial expressions modulate the time course of long latency auditory brain potentials

Long latency auditory brain potentials were recorded while subjects listened to bi-syllabic words spoken with an emotional expression and concurrently viewed congruent or incongruent facial expressions. Analysis of the auditory waveforms suggests the existence of a positive deflection around 240 ms post-stimulus with a clear posterior topography (the P2b component). This potential is subsequent upon the modality-specific auditory N1-P2 components and precedes the amodal N2-P3 complex. Congruent face-voice trials elicited an earlier P2b component than incongruent trials suggesting that auditory processing is delayed in the presence of an incongruent facial context. These electrophysiological results are consistent with previous behavioural studies showing an acceleration of reaction times for rating voice expressions that are part of congruent bimodal stimulus pairs. A source localisation analysis performed on the scalp EEG during the time-window corresponding to the P2b component disclosed a single dipole solution in the anterior cingulate cortex.

Spatial coincidence modulates interaction between visual and somatosensory evoked potentials

The time course of interaction between concurrently applied visual and somatosensory stimulation with respect to evoked potentials (EPs) was studied. Visual stimuli, either in the left or right hemifield, and electric stimuli to the left wrist were delivered either alone or simultaneously. Visual and somatosensory EPs were summed and compared to bimodal EPs (BiEP, response to actual combination of both modalities). Temporal coincidence of stimuli lead to sub-additive or over-additive amplitudes in BiEPs in several time windows between 75 and 275 ms. Additional effects of spatial coincidence (left wrist with left hemifield) were found between 75 and 300 ms and beyond 450 ms. These interaction effects hint at a temporo-spatial pattern of multiple brain areas participating in the process of multimodal integration.

Processing of changes in visual speech in the human auditory cortex

Seeing a talker's articulatory gestures may affect the observer's auditory speech percept. Observing congruent articulatory gestures may enhance the recognition of speech sounds [J. Acoust. Soc. Am. 26 (1954) 212], whereas observing incongruent gestures may change the auditory percept phonetically as occurs in the McGurk effect [Nature 264 (1976) 746]. For example, simultaneous acoustic /ba/ and visual /ga/ are usually heard as /da/. We studied cortical processing of occasional changes in audiovisual and visual speech stimuli with magnetoencephalography. In the audiovisual experiment congruent (acoustic /iti/, visual /iti/) and incongruent (acoustic /ipi/, visual /iti/) audiovisual stimuli, which were both perceived as /iti/, were presented among congruent /ipi/ (acoustic /ipi/, visual /ipi/) stimuli. In the visual experiment only the visual components of these stimuli were presented. A visual change both in audiovisual and visual experiments activated supratemporal auditory cortices bilaterally. The auditory cortex activation to a visual change occurred later in the visual than in the audiovisual experiment, suggesting that interaction between modalities accelerates the detection of visual change in speech.

The biological basis of speech: what to infer from talking to the animals

Speech perception and production are uniquely human adaptations. The mechanisms and laws that these adaptations implicate are tuned to linguistic rather than general auditory phenomena, leading to the view that speech is special (SiS). Despite the progress made by proponents of SiS, a small but growing "auditorist" program critical of SiS conscripts nonhuman animals such as quail and chinchilla and, using discrimination tasks on speech stimuli, incorrectly infers a common mechanism from similar cross-species performance. The author argues that the auditorist's refutation project must demonstrate not just cross-species isomorphisms of behavior but also either a common biological mechanism or common functional organization. The refutation project does neither. The author argues further that the auditorist's appeal to the principle of parsimony fails. Respect for the total available evidence undermines auditorism and bolsters SiS.

Cross-modal plasticity underpins language recovery after cochlear implantation

Postlingually deaf subjects learn the meaning of sounds after cochlear implantation by forming new associations between sounds and their sources. Implants generate coarse frequency responses, preventing place-coding fine enough to discriminate sounds with similar temporal characteristics, e.g., buck/duck. This limitation imposes a dependency on visual cues, e.g., lipreading. We hypothesized that cross-modal facilitation results from engagement of the visual cortex by purely auditory tasks. In four functional neuroimaging experiments, we show recruitment of early visual cortex (V1/V2) when cochlear implant users listen to sounds with eyes closed. Activity in visual cortex evolved in a stimulus-specific manner as a function of time from implantation reflecting experience-dependent adaptations in the postimplant phase.

The Morton-Massaro law of information integration: implications for models of perception

Information integration may be studied by analyzing the effect of 2 or more sources (e.g., auditory and visual) on participants' responses. Experiments show that ratios of response probabilities often factorize into components selectively influenced by only 1 source (e.g., 1 component affected by the acoustic source and another 1 affected by the visual source). This is called the Morton-Massaro law (MML). This article identifies conditions in which the law is optimal and notes that it reflects an implicit assumption about the statistics of the environment. Adherence to the MML can be used to assess whether the assumption is being made, and analyses of natural stimuli can be used to determine whether the assumption is reasonable. Feed-forward and interactive models subject to a channel separability constraint are consistent with the law.

Multimodal contribution to speech perception revealed by independent component analysis: a single-sweep EEG case study

In this single-sweep electroencephalographic case study, independent component analysis (ICA) was used to investigate multimodal processes underlying the enhancement of speech intelligibility in noise (for monosyllabic English words) by visualizing facial motion concordant with the audio speech signal. Wavelet analysis of the single-sweep IC activation waveforms revealed increased high-frequency energy for two ICs underlying the visual enhancement effect. For one IC, current source density analysis localized activity mainly to the superior temporal gyrus, consistent with principles of multimodal integration. For the other IC, activity was distributed across multiple cortical areas perhaps reflecting global mappings underlying the visual enhancement effect.

Timing of human cortical functions during cognition: role of MEG

Understanding of sensory and cognitive brain processes requires information about activation timing within and between different brain sites. Such data can be obtained by magnetoencephalography (MEG) that tracks cortical activation sequences with a millisecond temporal accuracy. MEG is gaining a well-established role in human neuroscience, complementing with its excellent temporal resolution the spatially more focused brain imaging methods. As examples of MEG's role in cognitive neuroscience, we discuss time windows related to cortical processing of sensory and multisensory stimuli, effects of the subject's own voice on the activity of their auditory cortex, timing of brain activation in reading, and cortical dynamics of the human mirror-neuron system activated when the subject views another person's movements.

Audiovisual integration of letters in the human brain

Letters of the alphabet have auditory (phonemic) and visual (graphemic) qualities. To investigate the neural representations of such audiovisual objects, we recorded neuromagnetic cortical responses to auditorily, visually, and audiovisually presented single letters. The auditory and visual brain activations first converged around 225 ms after stimulus onset and then interacted predominantly in the right temporo-occipito-parietal junction (280345 ms) and the left (380-540 ms) and right (450-535 ms) superior temporal sulci. These multisensory brain areas, playing a role in audiovisual integration of phonemes and graphemes, participate in the neural network supporting the supramodal concept of a "letter." The dynamics of these functions bring new insight into the interplay between sensory and association cortices during object recognition.

Multisensory perception: beyond modularity and convergence

Recent research on multisensory perception suggests a number of general principles for crossmodal integration and that the standard model in the field--feedforward convergence of information--must be modified to include a role for feedback projections from multimodal to unimodal brain areas.