Perceiving Biological Motion: Dissociating Visible Speech from Walking

Inhibitory TMS over Visual Area V5/MT Disrupts Visual Speech Recognition

During face-to-face communication, the perception and recognition of facial movements can facilitate individuals' understanding of what is said. Facial movements are a form of complex biological motion. Separate neural pathways are thought to processing (1) simple, nonbiological motion with an obligatory waypoint in the motion-sensitive visual middle temporal area (V5/MT); and (2) complex biological motion. Here, we present findings that challenge this dichotomy. Neuronavigated offline transcranial magnetic stimulation (TMS) over V5/MT on 24 participants (17 females and 7 males) led to increased response times in the recognition of simple, nonbiological motion as well as visual speech recognition compared with TMS over the vertex, an active control region. TMS of area V5/MT also reduced practice effects on response times, that are typically observed in both visual speech and motion recognition tasks over time. Our findings provide the first indication that area V5/MT causally influences the recognition of visual speech.SIGNIFICANCE STATEMENT In everyday face-to-face communication, speech comprehension is often facilitated by viewing a speaker's facial movements. Several brain areas contribute to the recognition of visual speech. One area of interest is the motion-sensitive visual medial temporal area (V5/MT), which has been associated with the perception of simple, nonbiological motion such as moving dots, as well as more complex, biological motion such as visual speech. Here, we demonstrate using noninvasive brain stimulation that area V5/MT is causally relevant in recognizing visual speech. This finding provides new insights into the neural mechanisms that support the perception of human communication signals, which will help guide future research in typically developed individuals and populations with communication difficulties.

Evaluation of gait termination strategy in individuals with Essential Tremor and Parkinson's disease

INTRODUCTION

Gait termination (GT) is a challenging transitory task involving converting from a dynamic state of motion to a static state. These transitional locomotor tasks are particularly troublesome for populations with postural deficits, i.e., Parkinson's disease (PD) and Essential Tremor (ET). They demand greater postural control and intricate integration of the neuromuscular system. The mechanisms involved in GT in these populations have not been well studied despite the safety concerns and potential risk for falls. The purpose of this investigation was to examine the different control strategies utilized during GT between individuals with ET and PD.

METHODS

Twenty-four individuals with ET (66 ± 8 yrs), twenty-four individuals with PD (64 ± 8 yrs), and twenty healthy older adults (HOA: 63 ± 9 yrs) participated in this study. Average self-selected gait velocity for each group was collected during the GT trial walking portion. Ground reaction force (GRF) data were used to calculate braking and propulsive forces from the last two steps during GT. GRF data measured the dynamic postural stability index (DPSI), defined as an individual's ability to maintain balance while transitioning from a dynamic to a stable state.

RESULTS

Persons with ET had a significantly slower approach velocity (0.63 m/s) when compared to HOA (0.92 m/s) and PD (0.77 m/s). Persons with PD had significantly slower approach velocity when compared to HOA. Examination of GRF data found that those with ET generated significantly smaller propulsive and braking forces (p < .05). Forces increased in those with PD and then even more in the HOA group. Postural stability analysis revealed that ET had significantly worse stability scores than PD and HOA (p < .05).

CONCLUSION

Individuals with PD and ET utilize different control strategies for planned GT, which suggests both the cerebellum and the basal ganglia play central yet potentially different roles in anticipatory control during self-directed activities.

Contrast Is in the Eye of the Beholder: Infelicitous Beat Gesture Increases Cognitive Load During Online Spoken Discourse Comprehension

We investigated how two cues to contrast-beat gesture and contrastive pitch accenting-affect comprehenders' cognitive load during processing of spoken referring expressions. In two visual-world experiments, we orthogonally manipulated the presence of these cues and their felicity, or fit, with the local (sentence-level) referential context in critical referring expressions while comprehenders' task-evoked pupillary responses (TEPRs) were examined. In Experiment 1, beat gesture and contrastive accenting always matched the referential context of filler referring expressions and were therefore relatively felicitous on the global (experiment) level, whereas in Experiment 2, beat gesture and contrastive accenting never fit the referential context of filler referring expressions and were therefore infelicitous on the global level. The results revealed that both beat gesture and contrastive accenting increased comprehenders' cognitive load. For beat gesture, this increase in cognitive load was driven by both local and global infelicity. For contrastive accenting, this increase in cognitive load was unaffected when cues were globally felicitous but exacerbated when cues were globally infelicitous. Together, these results suggest that comprehenders' cognitive resources are taxed by processing infelicitous use of beat gesture and contrastive accenting to convey contrast on both the local and global levels.

The interrelationship between the face and vocal tract configuration during audiovisual speech

Speech perception is improved when we are able to see the person who is speaking, but how visual speech cues are used to improve speech perception is currently unclear. Brain imaging has revealed that regions responsible for motor control are active during the perception of speech, opening up the possibility that visual cues are mapped onto an internal representation of the vocal tract. Here, we show that there is sufficient information in the configuration of the face to recover the vocal tract configuration and that the key areas responsible for driving the correspondence vary in accordance with the articulation required to form the acoustic signal at the appropriate point in a sentence.

It is well established that speech perception is improved when we are able to see the speaker talking along with hearing their voice, especially when the speech is noisy. While we have a good understanding of where speech integration occurs in the brain, it is unclear how visual and auditory cues are combined to improve speech perception. One suggestion is that integration can occur as both visual and auditory cues arise from a common generator: the vocal tract. Here, we investigate whether facial and vocal tract movements are linked during speech production by comparing videos of the face and fast magnetic resonance (MR) image sequences of the vocal tract. The joint variation in the face and vocal tract was extracted using an application of principal components analysis (PCA), and we demonstrate that MR image sequences can be reconstructed with high fidelity using only the facial video and PCA. Reconstruction fidelity was significantly higher when images from the two sequences corresponded in time, and including implicit temporal information by combining contiguous frames also led to a significant increase in fidelity. A “Bubbles” technique was used to identify which areas of the face were important for recovering information about the vocal tract, and vice versa, on a frame-by-frame basis. Our data reveal that there is sufficient information in the face to recover vocal tract shape during speech. In addition, the facial and vocal tract regions that are important for reconstruction are those that are used to generate the acoustic speech signal.

Sensitivity of occipito-temporal cortex, premotor and Broca's areas to visible speech gestures in a familiar language

When looking at a speaking person, the analysis of facial kinematics contributes to language discrimination and to the decoding of the time flow of visual speech. To disentangle these two factors, we investigated behavioural and fMRI responses to familiar and unfamiliar languages when observing speech gestures with natural or reversed kinematics. Twenty Italian volunteers viewed silent video-clips of speech shown as recorded (Forward, biological motion) or reversed in time (Backward, non-biological motion), in Italian (familiar language) or Arabic (non-familiar language). fMRI revealed that language (Italian/Arabic) and time-rendering (Forward/Backward) modulated distinct areas in the ventral occipito-temporal cortex, suggesting that visual speech analysis begins in this region, earlier than previously thought. Left premotor ventral (superior subdivision) and dorsal areas were preferentially activated with the familiar language independently of time-rendering, challenging the view that the role of these regions in speech processing is purely articulatory. The left premotor ventral region in the frontal operculum, thought to include part of the Broca's area, responded to the natural familiar language, consistent with the hypothesis of motor simulation of speech gestures.

The two-process theory of biological motion processing

Perception, identification, and understanding of others' actions from motion information are vital for our survival in the social world. A breakthrough in the understanding of action perception was the discovery that our visual system is sensitive to human action from the sparse motion input of only a dozen point lights, a phenomenon known as biological motion (BM) processing. Previous psychological and computational models cannot fully explain the emerging evidence for the existence of BM processing during early ontogeny. Here, we propose a two-process model of the mechanisms underlying BM processing. We hypothesize that the first system, the 'Step Detector,' rapidly processes the local foot motion and feet-below-the-body information that is specific to vertebrates, is less dependent on postnatal learning, and involves subcortical networks. The second system, the 'Bodily Action Evaluator,' slowly processes the fine global structure-from-motion, is specific to conspecific, and dependent on gradual learning processed in cortical networks. This proposed model provides new insight into research on the development of BM processing.

The Relationship between Biological Motion-Based Visual Consciousness and Attention: An Electroencephalograph Study

Understanding and predicting the intentions of others through limb movements are vital to social interaction. The processing of biological motion is unique from the processing of motion of inanimate objects. Presently, there is controversy over whether visual consciousness of biological motion is regulated by visual attention. In addition, the neural mechanisms involved in biological motion-related visual awareness are not known. In the current study, the relationship between visual awareness (aware vs unaware), represented by a point-light walker and biological-motion-based attention, manipulated by a difference in congruence (congruent, incongruent) between the direction of a pre-cue and that of biological motion was explored. The neural mechanisms involved in processing the stimuli were explored through electroencephalography. Both early (50-150 ms, 100-200 ms, and 174-226 ms after target presentation) and late (350-550 ms after target presentation) awareness-related neural processings were observed during a biological motion-based congruency task. Early processing was localized to occipital-parietal regions, such as the left postcentral gyrus, the left middle occipital gyrus, and the right precentral gyrus. In the 174-226-ms window, the activity in the occipital region was gradually replaced by activity in the parietal and frontal regions. Late processing was localized to frontal-parietal regions, such as the right dorsal superior frontal gyrus, the left medial superior frontal gyrus, and the occipito-temporal regions. Congruency-related processing occurred in the 246-260-ms window and was localized to the right superior occipital gyrus. In summary, due to its complexity, biological motion awareness has a unique neural basis.

Emotion modulation of the body-selective areas in the developing brain

•

Passive viewing fMRI task using dynamic emotional bodies and non-human objects.

•

Adults showed increased activation in the body-selective areas compared with children.

•

Adults also showed more activation than adolescents, but only in the right hemisphere.

•

Crucially, we found no age differences in the emotion modulation of these areas.

Emotions are strongly conveyed by the human body and the ability to recognize emotions from body posture or movement is still developing through childhood and adolescence. To date, very few studies have explored how these behavioural observations are paralleled by functional brain development. Furthermore, currently no studies have explored the development of emotion modulation in these areas. In this study, we used functional magnetic resonance imaging (fMRI) to compare the brain activity of 25 children (age 6–11), 18 adolescents (age 12–17) and 26 adults while they passively viewed short videos of angry, happy or neutral body movements. We observed that when viewing dynamic bodies generally, adults showed higher activity than children bilaterally in the body-selective areas; namely the extra-striate body area (EBA), fusiform body area (FBA), posterior superior temporal sulcus (pSTS), as well as the amygdala (AMY). Adults also showed higher activity than adolescents, but only in the right hemisphere. Crucially, however, we found that there were no age differences in the emotion modulation of activity in these areas. These results indicate, for the first time, that despite activity selective to body perception increasing across childhood and adolescence, emotion modulation of these areas in adult-like from 7 years of age.

Mouth and Voice: A Relationship between Visual and Auditory Preference in the Human Superior Temporal Sulcus

Cortex in and around the human posterior superior temporal sulcus (pSTS) is known to be critical for speech perception. The pSTS responds to both the visual modality (especially biological motion) and the auditory modality (especially human voices). Using fMRI in single subjects with no spatial smoothing, we show that visual and auditory selectivity are linked. Regions of the pSTS were identified that preferred visually presented moving mouths (presented in isolation or as part of a whole face) or moving eyes. Mouth-preferring regions responded strongly to voices and showed a significant preference for vocal compared with nonvocal sounds. In contrast, eye-preferring regions did not respond to either vocal or nonvocal sounds. The converse was also true: regions of the pSTS that showed a significant response to speech or preferred vocal to nonvocal sounds responded more strongly to visually presented mouths than eyes. These findings can be explained by environmental statistics. In natural environments, humans see visual mouth movements at the same time as they hear voices, while there is no auditory accompaniment to visual eye movements. The strength of a voxel's preference for visual mouth movements was strongly correlated with the magnitude of its auditory speech response and its preference for vocal sounds, suggesting that visual and auditory speech features are coded together in small populations of neurons within the pSTS.SIGNIFICANCE STATEMENT Humans interacting face to face make use of auditory cues from the talker's voice and visual cues from the talker's mouth to understand speech. The human posterior superior temporal sulcus (pSTS), a brain region known to be important for speech perception, is complex, with some regions responding to specific visual stimuli and others to specific auditory stimuli. Using BOLD fMRI, we show that the natural statistics of human speech, in which voices co-occur with mouth movements, are reflected in the neural architecture of the pSTS. Different pSTS regions prefer visually presented faces containing either a moving mouth or moving eyes, but only mouth-preferring regions respond strongly to voices.

Seeing biological actions in 3D: An fMRI study

Precise kinematics or body configuration cannot be recovered from visual input without disparity information. Yet, no imaging study has investigated the role of disparity on action observation. Here, we investigated the interaction between disparity and the main cues of biological motion, kinematics and configuration, in two fMRI experiments. Stimuli were presented as point‐light figures, depicting complex action sequences lasting 21 s. We hypothesized that interactions could occur at any of the three levels of the action observation network, comprising occipitotemporal, parietal and premotor cortex, with premotor cortex being the most likely location. The main effects of kinematics and configuration confirmed that the biological motion sequences activated all three levels of the action observation network, validating our approach. The interaction between configuration and disparity activated only premotor cortex, whereas interactions between kinematics and disparity occurred at all levels of the action observation network but were strongest at the premotor level. Control experiments demonstrated that these interactions could not be accounted for by low level motion in depth, task effects, spatial attention, or eye movements, including vergence. These results underscore the role of premotor cortex in action observation, and in imitating others or responding to their actions. Hum Brain Mapp 37:203–219, 2016. © 2015 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.

Social cognition and the cerebellum: A meta-analytic connectivity analysis

This meta-analytic connectivity modeling (MACM) study explores the functional connectivity of the cerebellum with the cerebrum in social cognitive processes. In a recent meta-analysis, Van Overwalle, Baetens, Mariën, and Vandekerckhove (2014) documented that the cerebellum is implicated in social processes of "body" reading (mirroring; e.g., understanding other persons' intentions from observing their movements) and "mind" reading (mentalizing, e.g., inferring other persons' beliefs, intentions or personality traits, reconstructing persons' past, future, or hypothetical events). In a recent functional connectivity study, Buckner et al. (2011) offered a novel parcellation of cerebellar topography that substantially overlaps with the cerebellar meta-analytic findings of Van Overwalle et al. (2014). This overlap suggests that the involvement of the cerebellum in social reasoning depends on its functional connectivity with the cerebrum. To test this hypothesis, we explored the meta-analytic co-activations as indices of functional connectivity between the cerebellum and the cerebrum during social cognition. The MACM results confirm substantial and distinct connectivity with respect to the functions of (a) action understanding ("body" reading) and (b) mentalizing ("mind" reading). The consistent and strong connectivity findings of this analysis suggest that cerebellar activity during social judgments reflects distinct mirroring and mentalizing functionality, and that these cerebellar functions are connected with corresponding functional networks in the cerebrum.

The contribution of dynamic visual cues to audiovisual speech perception

Seeing a speaker's facial gestures can significantly improve speech comprehension, especially in noisy environments. However, the nature of the visual information from the speaker's facial movements that is relevant for this enhancement is still unclear. Like auditory speech signals, visual speech signals unfold over time and contain both dynamic configural information and luminance-defined local motion cues; two information sources that are thought to engage anatomically and functionally separate visual systems. Whereas, some past studies have highlighted the importance of local, luminance-defined motion cues in audiovisual speech perception, the contribution of dynamic configural information signalling changes in form over time has not yet been assessed. We therefore attempted to single out the contribution of dynamic configural information to audiovisual speech processing. To this aim, we measured word identification performance in noise using unimodal auditory stimuli, and with audiovisual stimuli. In the audiovisual condition, speaking faces were presented as point light displays achieved via motion capture of the original talker. Point light displays could be isoluminant, to minimise the contribution of effective luminance-defined local motion information, or with added luminance contrast, allowing the combined effect of dynamic configural cues and local motion cues. Audiovisual enhancement was found in both the isoluminant and contrast-based luminance conditions compared to an auditory-only condition, demonstrating, for the first time the specific contribution of dynamic configural cues to audiovisual speech improvement. These findings imply that globally processed changes in a speaker's facial shape contribute significantly towards the perception of articulatory gestures and the analysis of audiovisual speech.

Increased Visual Stimulation Systematically Decreases Activity in Lateral Intermediate Cortex

Previous studies have attributed multiple diverse roles to the posterior superior temporal cortex (STC), both visually driven and cognitive, including part of the default mode network (DMN). Here, we demonstrate a unifying property across this multimodal region. Specifically, the lateral intermediate (LIM) portion of STC showed an unexpected feature: a progressively decreasing fMRI response to increases in visual stimulus size (or number). Such responses are reversed in sign, relative to well-known responses in classic occipital temporal visual cortex. In LIM, this "reversed" size function was present across multiple object categories and retinotopic eccentricities. Moreover, we found a significant interaction between the LIM size function and the distribution of subjects' attention. These findings suggest that LIM serves as a part of the DMN. Further analysis of functional connectivity, plus a meta-analysis of previous fMRI results, suggests that LIM is a heterogeneous area including different subdivisions. Surprisingly, analogous fMRI tests in macaque monkeys did not reveal a clear homolog of LIM. This interspecies discrepancy supports the idea that self-referential thinking and theory of mind are more prominent in humans, compared with monkeys.

Neural pathways for visual speech perception

This paper examines the questions, what levels of speech can be perceived visually, and how is visual speech represented by the brain? Review of the literature leads to the conclusions that every level of psycholinguistic speech structure (i.e., phonetic features, phonemes, syllables, words, and prosody) can be perceived visually, although individuals differ in their abilities to do so; and that there are visual modality-specific representations of speech qua speech in higher-level vision brain areas. That is, the visual system represents the modal patterns of visual speech. The suggestion that the auditory speech pathway receives and represents visual speech is examined in light of neuroimaging evidence on the auditory speech pathways. We outline the generally agreed-upon organization of the visual ventral and dorsal pathways and examine several types of visual processing that might be related to speech through those pathways, specifically, face and body, orthography, and sign language processing. In this context, we examine the visual speech processing literature, which reveals widespread diverse patterns of activity in posterior temporal cortices in response to visual speech stimuli. We outline a model of the visual and auditory speech pathways and make several suggestions: (1) The visual perception of speech relies on visual pathway representations of speech qua speech. (2) A proposed site of these representations, the temporal visual speech area (TVSA) has been demonstrated in posterior temporal cortex, ventral and posterior to multisensory posterior superior temporal sulcus (pSTS). (3) Given that visual speech has dynamic and configural features, its representations in feedforward visual pathways are expected to integrate these features, possibly in TVSA.

The development of sensorimotor influences in the audiovisual speech domain: some critical questions

Speech researchers have long been interested in how auditory and visual speech signals are integrated, and the recent work has revived interest in the role of speech production with respect to this process. Here, we discuss these issues from a developmental perspective. Because speech perception abilities typically outstrip speech production abilities in infancy and childhood, it is unclear how speech-like movements could influence audiovisual speech perception in development. While work on this question is still in its preliminary stages, there is nevertheless increasing evidence that sensorimotor processes (defined here as any motor or proprioceptive process related to orofacial movements) affect developmental audiovisual speech processing. We suggest three areas on which to focus in future research: (i) the relation between audiovisual speech perception and sensorimotor processes at birth, (ii) the pathways through which sensorimotor processes interact with audiovisual speech processing in infancy, and (iii) developmental change in sensorimotor pathways as speech production emerges in childhood.

Network Interactions Explain Sensitivity to Dynamic Faces in the Superior Temporal Sulcus

The superior temporal sulcus (STS) in the human and monkey is sensitive to the motion of complex forms such as facial and bodily actions. We used functional magnetic resonance imaging (fMRI) to explore network-level explanations for how the form and motion information in dynamic facial expressions might be combined in the human STS. Ventral occipitotemporal areas selective for facial form were localized in occipital and fusiform face areas (OFA and FFA), and motion sensitivity was localized in the more dorsal temporal area V5. We then tested various connectivity models that modeled communication between the ventral form and dorsal motion pathways. We show that facial form information modulated transmission of motion information from V5 to the STS, and that this face-selective modulation likely originated in OFA. This finding shows that form-selective motion sensitivity in the STS can be explained in terms of modulation of gain control on information flow in the motion pathway, and provides a substantial constraint for theories of the perception of faces and biological motion.

Individual differences in the perception of biological motion and fragmented figures are not correlated

We live in a cluttered, dynamic visual environment that poses a challenge for the visual system: for objects, including those that move about, to be perceived, information specifying those objects must be integrated over space and over time. Does a single, omnibus mechanism perform this grouping operation, or does grouping depend on separate processes specialized for different feature aspects of the object? To address this question, we tested a large group of healthy young adults on their abilities to perceive static fragmented figures embedded in noise and to perceive dynamic point-light biological motion figures embedded in dynamic noise. There were indeed substantial individual differences in performance on both tasks, but none of the statistical tests we applied to this data set uncovered a significant correlation between those performance measures. These results suggest that the two tasks, despite their superficial similarity, require different segmentation and grouping processes that are largely unrelated to one another. Whether those processes are embodied in distinct neural mechanisms remains an open question.

Social cognition and the cerebellum: a meta-analysis of over 350 fMRI studies

This meta-analysis explores the role of the cerebellum in social cognition. Recent meta-analyses of neuroimaging studies since 2008 demonstrate that the cerebellum is only marginally involved in social cognition and emotionality, with a few meta-analyses pointing to an involvement of at most 54% of the individual studies. In this study, novel meta-analyses of over 350 fMRI studies, dividing up the domain of social cognition in homogeneous subdomains, confirmed this low involvement of the cerebellum in conditions that trigger the mirror network (e.g., when familiar movements of body parts are observed) and the mentalizing network (when no moving body parts or unfamiliar movements are present). There is, however, one set of mentalizing conditions that strongly involve the cerebellum in 50-100% of the individual studies. In particular, when the level of abstraction is high, such as when behaviors are described in terms of traits or permanent characteristics, in terms of groups rather than individuals, in terms of the past (episodic autobiographic memory) or the future rather than the present, or in terms of hypothetical events that may happen. An activation likelihood estimation (ALE) meta-analysis conducted in this study reveals that the cerebellum is critically implicated in social cognition and that the areas of the cerebellum which are consistently involved in social cognitive processes show extensive overlap with the areas involved in sensorimotor (during mirror and self-judgments tasks) as well as in executive functioning (across all tasks). We discuss the role of the cerebellum in social cognition in general and in higher abstraction mentalizing in particular. We also point out a number of methodological limitations of some available studies on the social brain that hamper the detection of cerebellar activity.

The time course of auditory-visual processing of speech and body actions: evidence for the simultaneous activation of an extended neural network for semantic processing

An extensive network of cortical areas is involved in multisensory object and action recognition. This network draws on inferior frontal, posterior temporal, and parietal areas; activity is modulated by familiarity and the semantic congruency of auditory and visual component signals even if semantic incongruences are created by combining visual and auditory signals representing very different signal categories, such as speech and whole body actions. Here we present results from a high-density ERP study designed to examine the time-course and source location of responses to semantically congruent and incongruent audiovisual speech and body actions to explore whether the network involved in action recognition consists of a hierarchy of sequentially activated processing modules or a network of simultaneously active processing sites. We report two main results:1) There are no significant early differences in the processing of congruent and incongruent audiovisual action sequences. The earliest difference between congruent and incongruent audiovisual stimuli occurs between 240 and 280 ms after stimulus onset in the left temporal region. Between 340 and 420 ms, semantic congruence modulates responses in central and right frontal areas. Late differences (after 460 ms) occur bilaterally in frontal areas.2) Source localisation (dipole modelling and LORETA) reveals that an extended network encompassing inferior frontal, temporal, parasaggital, and superior parietal sites are simultaneously active between 180 and 420 ms to process auditory–visual action sequences. Early activation (before 120 ms) can be explained by activity in mainly sensory cortices. . The simultaneous activation of an extended network between 180 and 420 ms is consistent with models that posit parallel processing of complex action sequences in frontal, temporal and parietal areas rather than models that postulate hierarchical processing in a sequence of brain regions.

Developmental changes in point-light walker processing during childhood: a two-year follow-up ERP study

Event-related potentials were measured in twenty-four children aged 6-15 years, at one-year intervals for two years, to investigate developmental changes in each subject's neural response to a point-light walker (PLW) and a scrambled PLW (sPLW) stimulus. One positive peak (P1) and two negative peaks (N1 and N2) were observed in both occipitotemporal regions at approximately 130, 200, and 300-400ms. The amplitude and latency of the P1 component measured by the occipital electrode decreased during development over the first one-year period. Negative amplitudes of both N1 and N2, induced by the PLW stimulus, were significantly larger than those induced by the sPLW stimulus. Moreover, for the P1-N1 amplitude, the values for the eight-year-old children were significantly larger than those for the twelve-year-old children. N1 and N2 latency at certain electrodes decreased with age, but no consistent changes were observed. These results suggest that enhanced electrophysiological responses to PLW can be observed in all age groups, and that the early components were changed even over the course of a single year at the age of twelve.

The responsiveness of biological motion processing areas to selective attention towards goals

A growing literature indicates that visual cortex areas viewed as primarily responsive to exogenous stimuli are susceptible to top-down modulation by selective attention. The present study examines whether brain areas involved in biological motion perception are among these areas-particularly with respect to selective attention towards human movement goals. Fifteen participants completed a point-light biological motion study following a two-by-two factorial design, with one factor representing an exogenous manipulation of human movement goals (goal-directed versus random movement), and the other an endogenous manipulation (a goal identification task versus an ancillary color-change task). Both manipulations yielded increased activation in the human homologue of motion-sensitive area MT+ (hMT+) as well as the extrastriate body area (EBA). The endogenous manipulation was associated with increased right posterior superior temporal sulcus (STS) activation, whereas the exogenous manipulation was associated with increased activation in left posterior STS. Selective attention towards goals activated a portion of left hMT+/EBA only during the perception of purposeful movement-consistent with emerging theories associating this area with the matching of visual motion input to known goal-directed actions. The overall pattern of results indicates that attention towards the goals of human movement activates biological motion areas. Ultimately, selective attention may explain why some studies examining biological motion show activation in hMT+ and EBA, even when using control stimuli with comparable motion properties.

Dog experts' brains distinguish socially relevant body postures similarly in dogs and humans

We read conspecifics' social cues effortlessly, but little is known about our abilities to understand social gestures of other species. To investigate the neural underpinnings of such skills, we used functional magnetic resonance imaging to study the brain activity of experts and non-experts of dog behavior while they observed humans or dogs either interacting with, or facing away from a conspecific. The posterior superior temporal sulcus (pSTS) of both subject groups dissociated humans facing toward each other from humans facing away, and in dog experts, a distinction also occurred for dogs facing toward vs. away in a bilateral area extending from the pSTS to the inferior temporo-occipital cortex: the dissociation of dog behavior was significantly stronger in expert than control group. Furthermore, the control group had stronger pSTS responses to humans than dogs facing toward a conspecific, whereas in dog experts, the responses were of similar magnitude. These findings suggest that dog experts' brains distinguish socially relevant body postures similarly in dogs and humans.

Source cancellation profiles of electroencephalography and magnetoencephalography

Recorded electric potentials and magnetic fields due to cortical electrical activity have spatial spread even if their underlying brain sources are focal. Consequently, as a result of source cancellation, loss in signal amplitude and reduction in the effective signal-to-noise ratio can be expected when distributed sources are active simultaneously. Here we investigate the cancellation effects of EEG and MEG through the use of an anatomically correct forward model based on structural MRI acquired from 7 healthy adults. A boundary element model (BEM) with four compartments (brain, cerebrospinal fluid, skull and scalp) and highly accurate cortical meshes (~300,000 vertices) were generated. Distributed source activations were simulated using contiguous patches of active dipoles. To investigate cancellation effects in both EEG and MEG, quantitative indices were defined (source enhancement, cortical orientation disparity) and computed for varying values of the patch radius as well as for automatically parcellated gyri and sulci. Results were calculated for each cortical location, averaged over all subjects using a probabilistic atlas, and quantitatively compared between MEG and EEG. As expected, MEG sensors were found to be maximally sensitive to signals due to sources tangential to the scalp, and minimally sensitive to radial sources. Compared to EEG, however, MEG was found to be much more sensitive to signals generated antero-medially, notably in the anterior cingulate gyrus. Given that sources of activation cancel each other according to the orientation disparity of the cortex, this study provides useful methods and results for quantifying the effect of source orientation disparity upon source cancellation.

The perception of visible speech: estimation of speech rate and detection of time reversals

Four experiments investigated the perception of visible speech. Experiment 1 addressed the perception of speech rate. Observers were shown video-clips of the lower face of actors speaking at their spontaneous rate. Then, they were shown muted versions of the video-clips, which were either accelerated or decelerated. The task (scaling) was to compare visually the speech rate of the stimulus to the spontaneous rate of the actor being shown. Rate estimates were accurate when the video-clips were shown in the normal direction (forward mode). In contrast, speech rate was underestimated when the video-clips were shown in reverse (backward mode). Experiments 2-4 (2AFC) investigated how accurately one discriminates forward and backward speech movements. Unlike in Experiment 1, observers were never exposed to the sound track of the video-clips. Performance was well above chance when playback mode was crossed with rate modulation, and the number of repetitions of the stimuli allowed some amount of speechreading to take place in forward mode (Experiment 2). In Experiment 3, speechreading was made much more difficult by using a different and larger set of muted video-clips. Yet, accuracy decreased only slightly with respect to Experiment 2. Thus, kinematic rather then speechreading cues are most important for discriminating movement direction. Performance worsened, but remained above chance level when the same stimuli of Experiment 3 were rotated upside down (Experiment 4). We argue that the results are in keeping with the hypothesis that visual perception taps into implicit motor competence. Thus, lawful instances of biological movements (forward stimuli) are processed differently from backward stimuli representing movements that the observer cannot perform.

Modulation of the face- and body-selective visual regions by the motion and emotion of point-light face and body stimuli

Neural regions selective for facial or bodily form also respond to facial or bodily motion in highly form-degraded point-light displays. Yet it is unknown whether these face-selective and body-selective regions are sensitive to human motion regardless of stimulus type (faces and bodies) or to the specific motion-related cues characteristic of their proprietary stimulus categories. Using fMRI, we show that facial and bodily motions activate selectively those populations of neurons that code for the static structure of faces and bodies. Bodily (vs. facial) motion activated body-selective EBA bilaterally and right but not left FBA, irrespective of whether observers judged the emotion or color-change in point-light angry, happy and neutral stimuli. Facial (vs. bodily) motion activated face-selective right and left FFA, but only during emotion judgments for right FFA. Moreover, the strength of responses to point-light bodies vs. faces positively correlated with voxelwise selectivity for static bodies but not faces, whereas the strength of responses to point-light faces positively correlated with voxelwise selectivity for static faces but not bodies. Emotional content carried by point-light form-from-motion cues was sufficient to enhance the activity of several regions, including bilateral EBA and right FFA and FBA. However, although the strength of emotional modulation in right and left EBA by point-light body movements was related to the degree of voxelwise selectivity to static bodies but not static faces, there was no evidence that emotional modulation in fusiform cortex occurred in a similarly stimulus category-selective manner. This latter finding strongly constrains the claim that emotionally expressive movements modulate precisely those neuronal populations that code for the viewed stimulus category.

Detecting temporal reversals in human locomotion

An experiment investigated the ability by human observers to detect temporal reversals in dynamic displays of human locomotion. We video-taped the lower portion of the body of actors walking at their preferred speed either in the normal, forward direction (FW) or in the backward direction (BW). The videos were presented in a random order either as recorded (N) or in reverse (R). In one session, we presented both normal and time-reversed stimuli in the original upright orientation. In a second session, the stimuli were rotated by 180° around the horizontal axis. Observers were informed that the real recorded movement was either forward or backward and were asked to decide whether or not the movement had been time-reversed prior to the presentation. Although the kinematics of forward and backward human locomotion is quite similar, the detection of temporal reversals followed a consistent pattern showing a good accuracy in condition FW-N and a reduced but still above-chance performance in condition BW-R (by design, in both conditions actors appeared to walk forward). Performance was instead at chance level in the other two conditions where the apparent direction of the movement was backward. Inverting the spatial orientation of the stimuli reduced but did not suppress the ability to detect temporal reversals in the two conditions with apparent forward direction of movement. It is argued that implicit motor competence is at least in part instrumental for extracting the subtle discriminal information from the stimuli.

The signer and the sign: cortical correlates of person identity and language processing from point-light displays

In this study, the first to explore the cortical correlates of signed language (SL) processing under point-light display conditions, the observer identified either a signer or a lexical sign from a display in which different signers were seen producing a number of different individual signs. Many of the regions activated by point-light under these conditions replicated those previously reported for full-image displays, including regions within the inferior temporal cortex that are specialised for face and body-part identification, although such body parts were invisible in the display. Right frontal regions were also recruited – a pattern not usually seen in full-image SL processing. This activation may reflect the recruitment of information about person identity from the reduced display. A direct comparison of identify-signer and identify-sign conditions showed these tasks relied to a different extent on the posterior inferior regions. Signer identification elicited greater activation than sign identification in (bilateral) inferior temporal gyri (BA 37/19), fusiform gyri (BA 37), middle and posterior portions of the middle temporal gyri (BAs 37 and 19), and superior temporal gyri (BA 22 and 42). Right inferior frontal cortex was a further focus of differential activation (signer > sign).

These findings suggest that the neural systems supporting point-light displays for the processing of SL rely on a cortical network including areas of the inferior temporal cortex specialized for face and body identification. While this might be predicted from other studies of whole body point-light actions (Vaina, Solomon, Chowdhury, Sinha, & Belliveau, 2001) it is not predicted from the perspective of spoken language processing, where voice characteristics and speech content recruit distinct cortical regions (Stevens, 2004) in addition to a common network. In this respect, our findings contrast with studies of voice/speech recognition (Von Kriegstein, Kleinschmidt, Sterzer, & Giraud, 2005). Inferior temporal regions associated with the visual recognition of a person appear to be required during SL processing, for both carrier and content information.

Engagement of amygdala in third-person view of face-to-face interaction

Humans often watch interactions between other people without taking part in the interaction themselves. Strikingly little is, however, known about how gestures and expressions of two interacting humans are processed in the observer's brain, since the study of social cues has mostly focused on the perception of solitary humans. To investigate the neural underpinnings of the third-person view of social interaction, we studied brain activations of subjects who observed two humans either facing toward or away from each other. Activations within the amygdala, posterior superior temporal sulcus (pSTS), and dorsomedial prefrontal cortex (dmPFC) were sensitive to the interactional position of the observed people and distinguished humans facing toward from humans facing away. The amygdala was most sensitive to face-to-face interaction and did not differentiate the humans facing away from the pixelated control figures, whereas the pSTS dissociated both human stimuli from the pixel figures. The results of the amygdala reactivity suggest that, in addition to regulating interpersonal distance towards oneself, the amygdala is involved in the assessment of the proximity between two other persons.

Brain regions involved in human movement perception: a quantitative voxel-based meta-analysis

Face, hands, and body movements are powerful signals essential for social interactions. In the last 2 decades, a large number of brain imaging studies have explored the neural correlates of the perception of these signals. Formal synthesis is crucially needed, however, to extract the key circuits involved in human motion perception across the variety of paradigms and stimuli that have been used. Here, we used the activation likelihood estimation (ALE) meta-analysis approach with random effect analysis. We performed meta-analyses on three classes of biological motion: movement of the whole body, hands, and face. Additional analyses of studies of static faces or body stimuli and sub-analyses grouping experiments as a function of their control stimuli or task employed allowed us to identify main effects of movements and forms perception, as well as effects of task demand. In addition to specific features, all conditions showed convergence in occipito-temporal and fronto-parietal regions, but with different peak location and extent. The conjunction of the three ALE maps revealed convergence in all categories in a region of the right posterior superior temporal sulcus as well as in a bilateral region at the junction between middle temporal and lateral occipital gyri. Activation in these regions was not a function of attentional demand and was significant also when controlling for non-specific motion perception. This quantitative synthesis points towards a special role for posterior superior temporal sulcus for integrating human movement percept, and supports a specific representation for body parts in middle temporal, fusiform, precentral, and parietal areas.

The neurophysiology of human biological motion processing: a high-density electrical mapping study

The neural processing of biological motion (BM) is of profound experimental interest since it is often through the movement of another that we interpret their immediate intentions. Neuroimaging points to a specialized cortical network for processing biological motion. Here, high-density electrical mapping and source-analysis techniques were employed to interrogate the timing of information processing across this network. Participants viewed point-light-displays depicting standard body movements (e.g. jumping), while event-related potentials (ERPs) were recorded and compared to ERPs to scrambled motion control stimuli. In a pair of experiments, three major phases of BM-specific processing were identified: 1) The earliest phase of BM-sensitive modulation was characterized by a positive shift of the ERP between 100 and 200 ms after stimulus onset. This modulation was observed exclusively over the right hemisphere and source-analysis suggested a likely generator in close proximity to regions associated with general motion processing (KO/hMT). 2) The second phase of BM-sensitivity occurred from 200 to 350 ms, characterized by a robust negative-going ERP modulation over posterior middle temporal regions bilaterally. Source-analysis pointed to bilateral generators at or near the posterior superior temporal sulcus (STS). 3) A third phase of processing was evident only in our second experiment, where participants actively attended the BM aspect of the stimuli, and was manifest as a centro-parietal positive ERP deflection, likely related to later cognitive processes. These results point to very early sensory registration of biological motion, and highlight the interactive role of the posterior STS in analyzing the movements of other living organisms.

Interactions between auditory and visual semantic stimulus classes: evidence for common processing networks for speech and body actions

Incongruencies between auditory and visual signals negatively affect human performance and cause selective activation in neuroimaging studies; therefore, they are increasingly used to probe audiovisual integration mechanisms. An open question is whether the increased BOLD response reflects computational demands in integrating mismatching low-level signals or reflects simultaneous unimodal conceptual representations of the competing signals. To address this question, we explore the effect of semantic congruency within and across three signal categories (speech, body actions, and unfamiliar patterns) for signals with matched low-level statistics. In a localizer experiment, unimodal (auditory and visual) and bimodal stimuli were used to identify ROIs. All three semantic categories cause overlapping activation patterns. We find no evidence for areas that show greater BOLD response to bimodal stimuli than predicted by the sum of the two unimodal responses. Conjunction analysis of the unimodal responses in each category identifies a network including posterior temporal, inferior frontal, and premotor areas. Semantic congruency effects are measured in the main experiment. We find that incongruent combinations of two meaningful stimuli (speech and body actions) but not combinations of meaningful with meaningless stimuli lead to increased BOLD response in the posterior STS (pSTS) bilaterally, the left SMA, the inferior frontal gyrus, the inferior parietal lobule, and the anterior insula. These interactions are not seen in premotor areas. Our findings are consistent with the hypothesis that pSTS and frontal areas form a recognition network that combines sensory categorical representations (in pSTS) with action hypothesis generation in inferior frontal gyrus/premotor areas. We argue that the same neural networks process speech and body actions.

Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays

The talking face affords multiple types of information. To isolate cortical sites with responsibility for integrating linguistically relevant visual speech cues, speech and nonspeech face gestures were presented in natural video and point-light displays during fMRI scanning at 3.0T. Participants with normal hearing viewed the stimuli and also viewed localizers for the fusiform face area (FFA), the lateral occipital complex (LOC), and the visual motion (V5/MT) regions of interest (ROIs). The FFA, the LOC, and V5/MT were significantly less activated for speech relative to nonspeech and control stimuli. Distinct activation of the posterior superior temporal sulcus and the adjacent middle temporal gyrus to speech, independent of media, was obtained in group analyses. Individual analyses showed that speech and nonspeech stimuli were associated with adjacent but different activations, with the speech activations more anterior. We suggest that the speech activation area is the temporal visual speech area (TVSA), and that it can be localized with the combination of stimuli used in this study.

Representation of action in occipito-temporal cortex

A fundamental question for social cognitive neuroscience is how and where in the brain the identities and actions of others are represented. Here we present a replication and extension of a study by Kable and Chatterjee [Kable, J. W., & Chatterjee, A. Specificity of action representations in the lateral occipito-temporal cortex. Journal of Cognitive Neuroscience, 18, 1498-1517, 2006] examining the role of occipito-temporal cortex in these processes. We presented full-cue movies of actors performing whole-body actions and used fMRI to test for action- and identity-specific adaptation effects. We examined a series of functionally defined regions, including the extrastriate and fusiform body areas, the fusiform face area, the parahippocampal place area, the lateral occipital complex, the right posterior superior temporal sulcus, and motion-selective area hMT+. These regions were analyzed with both standard univariate measures as well as multivoxel pattern analyses. Additionally, we performed whole-brain tests for significant adaptation effects. We found significant action-specific adaptation in many areas, but no evidence for identity-specific adaptation. We argue that this finding could be explained by differences in the familiarity of the stimuli presented: The actions shown were familiar but the actors performing the actions were unfamiliar. However, in contrast to previous findings, we found that the action adaptation effect could not be conclusively tied to specific functionally defined regions. Instead, our results suggest that the adaptation to previously seen actions across identities is a widespread effect, evident across lateral and ventral occipito-temporal cortex.

Brain response to a humanoid robot in areas implicated in the perception of human emotional gestures

Background

The humanoid robot WE4-RII was designed to express human emotions in order to improve human-robot interaction. We can read the emotions depicted in its gestures, yet might utilize different neural processes than those used for reading the emotions in human agents.

Methodology

Here, fMRI was used to assess how brain areas activated by the perception of human basic emotions (facial expression of Anger, Joy, Disgust) and silent speech respond to a humanoid robot impersonating the same emotions, while participants were instructed to attend either to the emotion or to the motion depicted.

Principal Findings

Increased responses to robot compared to human stimuli in the occipital and posterior temporal cortices suggest additional visual processing when perceiving a mechanical anthropomorphic agent. In contrast, activity in cortical areas endowed with mirror properties, like left Broca's area for the perception of speech, and in the processing of emotions like the left anterior insula for the perception of disgust and the orbitofrontal cortex for the perception of anger, is reduced for robot stimuli, suggesting lesser resonance with the mechanical agent. Finally, instructions to explicitly attend to the emotion significantly increased response to robot, but not human facial expressions in the anterior part of the left inferior frontal gyrus, a neural marker of motor resonance.

Conclusions

Motor resonance towards a humanoid robot, but not a human, display of facial emotion is increased when attention is directed towards judging emotions.

Significance

Artificial agents can be used to assess how factors like anthropomorphism affect neural response to the perception of human actions.

A multisensory cortical network for understanding speech in noise

In noisy environments, listeners tend to hear a speaker's voice yet struggle to understand what is said. The most effective way to improve intelligibility in such conditions is to watch the speaker's mouth movements. Here we identify the neural networks that distinguish understanding from merely hearing speech, and determine how the brain applies visual information to improve intelligibility. Using functional magnetic resonance imaging, we show that understanding speech-in-noise is supported by a network of brain areas including the left superior parietal lobule, the motor/premotor cortex, and the left anterior superior temporal sulcus (STS), a likely apex of the acoustic processing hierarchy. Multisensory integration likely improves comprehension through improved communication between the left temporal-occipital boundary, the left medial-temporal lobe, and the left STS. This demonstrates how the brain uses information from multiple modalities to improve speech comprehension in naturalistic, acoustically adverse conditions.

Human functional magnetic resonance imaging reveals separation and integration of shape and motion cues in biological motion processing

In a series of human functional magnetic resonance imaging experiments, we systematically manipulated point-light stimuli to identify the contributions of the various areas implicated in biological motion processing (for review, see Giese and Poggio, 2003). The first experiment consisted of a 2 x 2 factorial design with global shape and kinematics as factors. In two additional experiments, we investigated the contributions of local opponent motion, the complexity of the portrayed movement and a one-back task to the activation pattern. Experiment 1 revealed a clear separation between shape and motion processing, resulting in two branches of activation. A ventral region, extending from the lateral occipital sulcus to the posterior inferior temporal gyrus, showed a main effect of shape and its extension into the fusiform gyrus also an interaction. The dorsal region, including the posterior inferior temporal sulcus and the posterior superior temporal sulcus (pSTS), showed a main effect of kinematics together with an interaction. Region of interest analysis identified these interaction sites as the extrastriate and fusiform body areas (EBA and FBA). The local opponent motion cue yielded only little activation, limited to the ventral region (experiment 3). Our results suggest that the EBA and the FBA correspond to the initial stages in visual action analysis, in which the performed action is linked to the body of the actor. Moreover, experiment 2 indicates that the body areas are activated automatically even in the absence of a task, whereas other cortical areas like pSTS or frontal regions depend on the complexity of movements or task instructions for their activation.

Mismatch negativity with visual-only and audiovisual speech

The functional organization of cortical speech processing is thought to be hierarchical, increasing in complexity and proceeding from primary sensory areas centrifugally. The current study used the mismatch negativity (MMN) obtained with electrophysiology (EEG) to investigate the early latency period of visual speech processing under both visual-only (VO) and audiovisual (AV) conditions. Current density reconstruction (CDR) methods were used to model the cortical MMN generator locations. MMNs were obtained with VO and AV speech stimuli at early latencies (approximately 82-87 ms peak in time waveforms relative to the acoustic onset) and in regions of the right lateral temporal and parietal cortices. Latencies were consistent with bottom-up processing of the visible stimuli. We suggest that a visual pathway extracts phonetic cues from visible speech, and that previously reported effects of AV speech in classical early auditory areas, given later reported latencies, could be attributable to modulatory feedback from visual phonetic processing.

Asymmetries of the human social brain in the visual, auditory and chemical modalities

Structural and functional asymmetries are present in many regions of the human brain responsible for motor control, sensory and cognitive functions and communication. Here, we focus on hemispheric asymmetries underlying the domain of social perception, broadly conceived as the analysis of information about other individuals based on acoustic, visual and chemical signals. By means of these cues the brain establishes the border between 'self' and 'other', and interprets the surrounding social world in terms of the physical and behavioural characteristics of conspecifics essential for impression formation and for creating bonds and relationships. We show that, considered from the standpoint of single- and multi-modal sensory analysis, the neural substrates of the perception of voices, faces, gestures, smells and pheromones, as evidenced by modern neuroimaging techniques, are characterized by a general pattern of right-hemispheric functional asymmetry that might benefit from other aspects of hemispheric lateralization rather than constituting a true specialization for social information.

Developmental changes in point-light walker processing during childhood and adolescence: an event-related potential study

To investigate developmental changes in the neural responses to a biological motion stimulus, we measured event-related potentials (ERPs) in 50 children aged from 7 to 14 years, and 10 adults. Two kinds of visual stimuli were presented: a point-light walker (PLW) stimulus and a scrambled point-light walker (sPLW) stimulus as a control. The sPLW stimulus had the same number of point-lights and the same velocity vector of point-lights as the PLW stimulus, but the initial starting positions were randomized. Consistent with previous ERP studies, one positive peak (P1) and two negative peaks (N1 and N2) were observed at around 130, 200 and 330 ms, respectively, in bilateral occipitotemporal regions, in all age groups. The latency of the P1 component was significantly shorter for the PLW than sPLW stimulus in all age groups, whereas the amplitude was significantly larger for the PLW than sPLW stimulus only for the 7-year-old group. The P1 amplitude and N1 latency were linearly decreased with age. The negative amplitudes of both N1 and N2 components of the PLW stimulus were significantly larger than those of the sPLW stimulus in all age groups. P1-N1 amplitude was changed by development, but not N2 amplitude. These results suggest that the intensity (P1) and timing (N1) of early visual processing for the PLW stimulus changed linearly throughout childhood and P1-N1 amplitude at occipitotemporal electrodes and N1 latency in 10-year-olds, but not 11-year-olds, was significantly larger than that in adults. For the amplitudes of the N2 component in response to PLW and sPLW stimuli in 7-8-year-old subjects were not statistically different from those in adults at occipitotemporal electrodes. These results suggest that the neural response to the PLW stimulus has developed by 10 years of age at the occipitotemporal electrode.

The role of spatial and temporal information in biological motion perception

Point-light biological motion stimuli provide spatio-temporal information about the structure of the human body in motion. Manipulation of the spatial structure of point-light stimuli reduces the ability of human observers to perceive biological motion. A recent study has reported that interference with the spatial structure of pointlight walkers also reduces the evoked eventrelated potentials over the occipitotemporal cortex, but that interference with the temporal structure of the stimuli evoked event-related potentials similar to normal biological motion stimuli. We systematically investigated the influence of spatial and temporal manipulation on 2 common discrimination tasks and compared it with predictions of a neurocomputational model previously proposed. This model first analyzes the spatial structure of the stimulus independently of the temporal information to derive body posture and subsequently analyzes the temporal sequence of body postures to derive movement direction. Similar to the model predictions, the psychophysical results show that human observers need only intact spatial configuration of the stimulus to discriminate the facing direction of a point-light walker. In contrast, movement direction discrimination needs a fully intact spatiotemporal pattern of the stimulus. The activation levels in the model predict the observed eventrelated potentials for the spatial and temporal manipulations.

The processing of audio-visual speech: empirical and neural bases

In this selective review, I outline a number of ways in which seeing the talker affects auditory perception of speech, including, but not confined to, the McGurk effect. To date, studies suggest that all linguistic levels are susceptible to visual influence, and that two main modes of processing can be described: a complementary mode, whereby vision provides information more efficiently than hearing for some under-specified parts of the speech stream, and a correlated mode, whereby vision partially duplicates information about dynamic articulatory patterning.Cortical correlates of seen speech suggest that at the neurological as well as the perceptual level, auditory processing of speech is affected by vision, so that 'auditory speech regions' are activated by seen speech. The processing of natural speech, whether it is heard, seen or heard and seen, activates the perisylvian language regions (left>right). It is highly probable that activation occurs in a specific order. First, superior temporal, then inferior parietal and finally inferior frontal regions (left>right) are activated. There is some differentiation of the visual input stream to the core perisylvian language system, suggesting that complementary seen speech information makes special use of the visual ventral processing stream, while for correlated visual speech, the dorsal processing stream, which is sensitive to visual movement, may be relatively more involved.