Premotor Cortex Is Sensitive to Auditory–Visual Congruence for Biological Motion

Enhancing learning outcomes through multisensory integration: A fMRI study of audio-visual training in virtual reality

The integration of information from different sensory modalities is a fundamental process that enhances perception and performance in real and virtual environments (VR). Understanding these mechanisms, especially during learning tasks that exploit novel multisensory cue combinations provides opportunities for the development of new rehabilitative interventions. This study aimed to investigate how functional brain changes support behavioural performance improvements during an audio-visual (AV) learning task. Twenty healthy participants underwent a 30 min daily VR training for four weeks. The task was an AV adaptation of a 'scanning training' paradigm that is commonly used in hemianopia rehabilitation. Functional magnetic resonance imaging (fMRI) and performance data were collected at baseline, after two and four weeks of training, and four weeks post-training. We show that behavioural performance, operationalised as mean reaction time reduction in VR, significantly improves. In separate tests in a controlled laboratory environment, we showed that the behavioural performance gains in the VR training environment transferred to a significant mean RT reduction for the trained AV voluntary task on a computer screen. Enhancements were observed in both the visual-only and AV conditions, with the latter demonstrating a faster response time supported by the presence of audio cues. The behavioural learning effect also transfers to two additional tasks that were tested: a visual search task and an involuntary visual task. Our fMRI results reveal an increase in functional activation (BOLD signal) in multisensory brain regions involved in early-stage AV processing: the thalamus, the caudal inferior parietal lobe and cerebellum. These functional changes were only observed for the trained, multisensory, task and not for unimodal visual stimulation. Functional activation changes in the thalamus were significantly correlated to behavioural performance improvements. This study demonstrates that incorporating spatial auditory cues to voluntary visual training in VR leads to augmented brain activation changes in multisensory integration, resulting in measurable performance gains across tasks. The findings highlight the potential of VR-based multisensory training as an effective method for enhancing cognitive function and as a potentially valuable tool in rehabilitative programmes.

Audiovisual correspondence facilitates the visual search for biological motion

Hearing synchronous sounds may facilitate the visual search for the concurrently changed visual targets. Evidence for this audiovisual attentional facilitation effect mainly comes from studies using artificial stimuli with relatively simple temporal dynamics, indicating a stimulus-driven mechanism whereby synchronous audiovisual cues create a salient object to capture attention. Here, we investigated the crossmodal attentional facilitation effect on biological motion (BM), a natural, biologically significant stimulus with complex and unique dynamic profiles. We found that listening to temporally congruent sounds, compared with incongruent sounds, enhanced the visual search for BM targets. More intriguingly, such a facilitation effect requires the presence of distinctive local motion cues (especially the accelerations in feet movement) independent of the global BM configuration, suggesting a crossmodal mechanism triggered by specific biological features to enhance the salience of BM signals. These findings provide novel insights into how audiovisual integration boosts attention to biologically relevant motion stimuli and extend the function of a proposed life detection system driven by local kinematics of BM to multisensory life motion perception.

Neural Correlates of Handwriting Effects in L2 Learners

Learning to write involves integrating motor production and visual perception to develop orthographic representations. This study tries to test the effect of hand movement training as a pathway to neural correlates for L2 Chinese and L2 English readers. Twenty L2 Chinese and 20 L2 English (n = 20) adults participated in both behavioral and electroencephalogram (EEG) experiments. We designed six learning conditions: Hand Writing Chinese (HC), Viewing Chinese (VC), Drawing followed by Character Recognition in Chinese (DC), Hand Writing English (HE), Viewing English (VE), and Drawing followed by Word Recognition in English (DE). Behavioral and EEG results demonstrated that drawing facilitated visual word recognition in Chinese compared to viewing. The findings imply that hand movement could strengthen the neural processing and improve behavioral performance in Chinese character recognition for L2 Chinese learners and English word recognition for L2 Chinese learners. Furthermore, N170 amplitude at the drawing condition was positively correlated with N400 amplitudes. Thus, the early visual word recognition neural indicator (e.g., N170) was predictive of the late neural indicator of semantic processing (e.g., N400), suggesting that hand movement facilitates the neural correlates between early word recognition and later comprehension.

Curved vs. Straight-Line Handwriting Effects on Word Recognition in Typical and Dyslexic Readers Across Chinese and English

Handwriting serves to link auditory and motor routines with visual word processing, which is a hallmark of successful reading. The current study aims to explore the effect of multisensory integration as a pathway to neural specialization for print among typical and dyslexic readers across writing systems. We identified 9-10-year-old dyslexic Chinese children (n = 24) and their typically developing counterparts (n = 24) on whom we conducted both behavioral and electroencephalogram (EEG) experiments. We designed four learning conditions: Handwriting Chinese (HC), Viewing Chinese (VC), Drawing followed by Character Recognition in Chinese (D-C), and Drawing followed by Word Recognition in English (D-E). In both handwriting and drawing conditions, we also designed curved vs. straight-line stimuli. Both behavioral and EEG results showed that handwriting straight line strokes facilitated visual word recognition in Chinese compared to handwriting curved lines. Handwriting conditions resulted in a lateralization of the N170 in typical readers, but not the dyslexic readers. Interestingly, drawing curved lines facilitate word recognition in English among dyslexic readers. Taken together, the results of the study suggest benefits of handwriting on the neural processing and behavioral performance in response to Chinese character recognition and curved-line drawing effects on English word recognition among dyslexic readers. But the lack of handwriting effects in dyslexic readers suggest that students who have deficits in reading may also be missing the link between multisensory integration and word recognition in the visual word form areas. The current study results have implications for maintaining handwriting practices to promote perception and motor integration for visual word form area development for normal readers and suggest that drawing practices might benefit Chinese dyslexic readers in reading English.

More than just statics: Temporal dynamic changes of intrinsic brain activity in cigarette smoking

Smoking is companied with altered intrinsic activity of the brain measured by amplitude of low-frequency fluctuation. Evidence has revealed that human brain activity is a highly dynamic and rapidly changing system. How exactly cigarette smoking affect temporal dynamic intrinsic brain activity is not fully understood nor is it clear how smoking severity influences spontaneous brain activity. Dynamic amplitude of low-frequency fluctuation (dALFF) was used to examine the dynamic temporal variability in 93 participants (63 smokers, 30 nonsmokers). We further divided smokers into light and heavy smokers. The temporal variability in intrinsic brain activity among these groups was compared. Correlation analyses were performed between dALFF in areas showing group differences and smoking behaviour (e.g., the Fagerström Test for Nicotine Dependence [FTND] scores and pack-years). Smokers showed significantly increased dALFF in the left inferior/middle frontal gyrus, right orbitofrontal gyrus, right insula, left superior/medial frontal gyrus and right middle frontal gyrus than nonsmokers. Light smokers showed increased dALFF variability in the left prefrontal cortex. Heavy smokers showed increased dynamics in specific brain regions, including the right postcentral gyrus, right insula and left precentral gyrus. Furthermore, the temporal variability in dALFF in the left superior/medial frontal gyrus, left superior/middle frontal gyrus, right middle frontal gyrus and right insula was positively correlated with pack-years or FTND. Combined, these results suggest that smokers increase stable and persistent spontaneous brain activity in prefrontal cortex, involved impaired gold-directed action and value-based decision-making. In addition, individuals with heavier smoking severity show increased perturbance on spontaneous brain activity of perception and sensorimotor, related to increased reliance.

Effect of Handwriting on Visual Word Recognition in Chinese Bilingual Children and Adults

In a digital era that neglects handwriting, the current study is significant because it examines the mechanisms underlying this process. We recruited 9- to 10-year-old Chinese children (n = 24), who were at an important period of handwriting development, and adult college students (n = 24), for both behavioral and electroencephalogram (EEG) experiments. We designed four learning conditions: handwriting Chinese (HC), viewing Chinese (VC), drawing shapes followed by Chinese recognition (DC), and drawing shapes followed by English recognition (DE). Both behavioral and EEG results showed that HC facilitated visual word recognition compared to VC, and behavioral results showed that HC facilitated visual word recognition compared to drawing shapes. HC and VC resulted in a lateralization of the N170 in adults, but not in children. Taken together, the results of the study suggest benefits of handwriting on the neural processing and behavioral performance in response to Chinese characters. The study results argue for maintaining handwriting practices to promote the perception of visual word forms in the digital age.

Visual Neuropsychology in Development: Anatomo-Functional Brain Mechanisms of Action/Perception Binding in Health and Disease

Vision is the main entrance for environmental input to the human brain. Even if vision is our most used sensory modality, its importance is not limited to environmental exploration. Rather it has strong links to motor competences, further extending to cognitive and social aspects of human life. These multifaceted relationships are particularly important in developmental age and become dramatically evident in presence of complex deficits originating from visual aberrancies. The present review summarizes the available neuropsychological evidence on the development of visual competences, with a particular focus on the associated visuo-motor integration skills in health and disease. With the aim of supporting future research and interventional settings, the goal of the present review is to constitute a solid base to help the translation of neuropsychological hypotheses into straightforward empirical investigations and rehabilitation/training protocols. This approach will further increase the impact, ameliorate the acceptance, and ease the use and implementation of lab-derived intervention protocols in real-life situations.

Auditory cues facilitate object movement processing in human extrastriate visual cortex during simulated self-motion: A pilot study

Visual segregation of moving objects is a considerable computational challenge when the observer moves through space. Recent psychophysical studies suggest that directionally congruent, moving auditory cues can substantially improve parsing object motion in such settings, but the exact brain mechanisms and visual processing stages that mediate these effects are still incompletely known. Here, we utilized multivariate pattern analyses (MVPA) of MRI-informed magnetoencephalography (MEG) source estimates to examine how crossmodal auditory cues facilitate motion detection during the observer's self-motion. During MEG recordings, participants identified a target object that moved either forward or backward within a visual scene that included nine identically textured objects simulating forward observer translation. Auditory motion cues 1) improved the behavioral accuracy of target localization, 2) significantly modulated the MEG source activity in the areas V2 and human middle temporal complex (hMT+), and 3) increased the accuracy at which the target movement direction could be decoded from hMT+ activity using MVPA. The increase of decoding accuracy by auditory cues in hMT+ was significant also when superior temporal activations in or near auditory cortices were regressed out from the hMT+ source activity to control for source estimation biases caused by point spread. Taken together, these results suggest that parsing object motion from self-motion-induced optic flow in the human extrastriate visual cortex can be facilitated by crossmodal influences from auditory system.

Auditory and Visual Motion Processing and Integration in the Primate Cerebral Cortex

The ability of animals to detect motion is critical for survival, and errors or even delays in motion perception may prove costly. In the natural world, moving objects in the visual field often produce concurrent sounds. Thus, it can highly advantageous to detect motion elicited from sensory signals of either modality, and to integrate them to produce more reliable motion perception. A great deal of progress has been made in understanding how visual motion perception is governed by the activity of single neurons in the primate cerebral cortex, but far less progress has been made in understanding both auditory motion and audiovisual motion integration. Here we, review the key cortical regions for motion processing, focussing on translational motion. We compare the representations of space and motion in the visual and auditory systems, and examine how single neurons in these two sensory systems encode the direction of motion. We also discuss the way in which humans integrate of audio and visual motion cues, and the regions of the cortex that may mediate this process.

Healthy young adults implement distinctive avoidance strategies while walking and circumventing virtual human vs. non-human obstacles in a virtual environment

This study examines how three types of obstacles (cylinder, virtual human and virtual human with footstep sounds) affect circumvention strategies of healthy young adults. Sixteen participants aged 25.2 ± 2.5 years (mean ± 1SD) were tested while walking overground and viewing a virtual room through a helmet mounted display. As participants walked towards a stationary target in the far space, they avoided an obstacle (cylinder or virtual human) approaching either from the right (+40°), left (-40°) or head-on (0°). Obstacle avoidance strategies were characterized using the position and orientation of the head. Repeated mixed model analysis showed smaller minimal distances (p = 0.007) while avoiding virtual humans as compared to cylinders. Footstep sounds added to virtual humans did not modify (p = 0.2) minimal distances compared to when no sound was provided. Onset times of avoidance strategies were similar across conditions (p = 0.06). Results indicate that the nature of the obstacle (human-like vs. non-human object) matters and can modify avoidance strategies. Smaller obstacle clearances in response to virtual humans may reflect the use of a less conservative avoidance strategy, due to a resemblance of obstacles to pedestrians and a recall of strategies used in daily locomotion. The lack of influence of footstep sounds supports the fact that obstacle avoidance primarily relies on visual cues and the principle of 'inverse effectiveness' whereby multisensory neurons' response to multimodal stimuli becomes weaker when the unimodal sensory stimulus (vision) is strong. Present findings should be taken into consideration to optimize the ecological validity of VR-based obstacle avoidance paradigms used in research and rehabilitation.

Effects of Motion in the Far Peripheral Visual Field on Cognitive Test Performance and Cognitive Load

Cognitive load theory posits that limited attention is in actuality a limitation in working memory resources. The load theory of selective attention and cognitive control sees the interplay between attention and awareness as separate modifying functions that act on working memory. Reconciling the theoretical differences in these two theories has important implications for learning. Thirty-nine adult participants performed a cognitively demanding test, with and without movement in the far peripheral field. Although the results for movement effects on cognitive load in this experiment were not statistically significant, men spent less time on the cognitive test in the peripheral movement condition than in the conditions without peripheral movement. No such difference was found for women. The implications of these results and recommendations for future research that extends the present study are presented.

Amplitude-modulated stimuli reveal auditory-visual interactions in brain activity and brain connectivity

The temporal congruence between auditory and visual signals coming from the same source can be a powerful means by which the brain integrates information from different senses. To investigate how the brain uses temporal information to integrate auditory and visual information from continuous yet unfamiliar stimuli, we used amplitude-modulated tones and size-modulated shapes with which we could manipulate the temporal congruence between the sensory signals. These signals were independently modulated at a slow or a fast rate. Participants were presented with auditory-only, visual-only, or auditory-visual (AV) trials in the fMRI scanner. On AV trials, the auditory and visual signal could have the same (AV congruent) or different modulation rates (AV incongruent). Using psychophysiological interaction analyses, we found that auditory regions showed increased functional connectivity predominantly with frontal regions for AV incongruent relative to AV congruent stimuli. We further found that superior temporal regions, shown previously to integrate auditory and visual signals, showed increased connectivity with frontal and parietal regions for the same contrast. Our findings provide evidence that both activity in a network of brain regions and their connectivity are important for AV integration, and help to bridge the gap between transient and familiar AV stimuli used in previous studies.

The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect

The Colavita effect refers to the phenomenon that when confronted with an audiovisual stimulus, observers report more often to have perceived the visual than the auditory component. The Colavita effect depends on low-level stimulus factors such as spatial and temporal proximity between the unimodal signals. Here, we examined whether the Colavita effect is modulated by synesthetic congruency between visual size and auditory pitch. If the Colavita effect depends on synesthetic congruency, we expect a larger Colavita effect for synesthetically congruent size/pitch (large visual stimulus/low-pitched tone; small visual stimulus/high-pitched tone) than synesthetically incongruent (large visual stimulus/high-pitched tone; small visual stimulus/low-pitched tone) combinations. Participants had to identify stimulus type (visual, auditory or audiovisual). The study replicated the Colavita effect because participants reported more often the visual than auditory component of the audiovisual stimuli. Synesthetic congruency had, however, no effect on the magnitude of the Colavita effect. EEG recordings to congruent and incongruent audiovisual pairings showed a late frontal congruency effect at 400-550 ms and an occipitoparietal effect at 690-800 ms with neural sources in the anterior cingulate and premotor cortex for the 400- to 550-ms window and premotor cortex, inferior parietal lobule and the posterior middle temporal gyrus for the 690- to 800-ms window. The electrophysiological data show that synesthetic congruency was probably detected in a processing stage subsequent to the Colavita effect. We conclude that-in a modality detection task-the Colavita effect can be modulated by low-level structural factors but not by higher-order associations between auditory and visual inputs.

Inferring common cognitive mechanisms from brain blood-flow lateralization data: a new methodology for fTCD analysis

Current neuroimaging techniques with high spatial resolution constrain participant motion so that many natural tasks cannot be carried out. The aim of this paper is to show how a time-locked correlation-analysis of cerebral blood flow velocity (CBFV) lateralization data, obtained with functional TransCranial Doppler (fTCD) ultrasound, can be used to infer cerebral activation patterns across tasks. In a first experiment we demonstrate that the proposed analysis method results in data that are comparable with the standard Lateralization Index (LI) for within-task comparisons of CBFV patterns, recorded during cued word generation (CWG) at two difficulty levels. In the main experiment we demonstrate that the proposed analysis method shows correlated blood-flow patterns for two different cognitive tasks that are known to draw on common brain areas, CWG, and Music Synthesis. We show that CBFV patterns for Music and CWG are correlated only for participants with prior musical training. CBFV patterns for tasks that draw on distinct brain areas, the Tower of London and CWG, are not correlated. The proposed methodology extends conventional fTCD analysis by including temporal information in the analysis of cerebral blood-flow patterns to provide a robust, non-invasive method to infer whether common brain areas are used in different cognitive tasks. It complements conventional high resolution imaging techniques.

Peak velocity as a cue in audiovisual synchrony perception of rhythmic stimuli

This study investigated audiovisual synchrony perception in a rhythmic context, where the sound was not consequent upon the observed movement. Participants judged synchrony between a bouncing point-light figure and an auditory rhythm in two experiments. Two questions were of interest: (1) whether the reference in the visual movement, with which the auditory beat should coincide, relies on a position or a velocity cue; (2) whether the figure form and motion profile affect synchrony perception. Experiment 1 required synchrony judgment with regard to the same (lowest) position of the movement in four visual conditions: two figure forms (human or non-human) combined with two motion profiles (human or ball trajectory). Whereas figure form did not affect synchrony perception, the point of subjective simultaneity differed between the two motions, suggesting that participants adopted the peak velocity in each downward trajectory as their visual reference. Experiment 2 further demonstrated that, when judgment was required with regard to the highest position, the maximal synchrony response was considerably low for ball motion, which lacked a peak velocity in the upward trajectory. The finding of peak velocity as a cue parallels results of visuomotor synchronization tasks employing biological stimuli, suggesting that synchrony judgment with rhythmic motions relies on the perceived visual beat.

The time course of auditory-visual processing of speech and body actions: evidence for the simultaneous activation of an extended neural network for semantic processing

An extensive network of cortical areas is involved in multisensory object and action recognition. This network draws on inferior frontal, posterior temporal, and parietal areas; activity is modulated by familiarity and the semantic congruency of auditory and visual component signals even if semantic incongruences are created by combining visual and auditory signals representing very different signal categories, such as speech and whole body actions. Here we present results from a high-density ERP study designed to examine the time-course and source location of responses to semantically congruent and incongruent audiovisual speech and body actions to explore whether the network involved in action recognition consists of a hierarchy of sequentially activated processing modules or a network of simultaneously active processing sites. We report two main results:1) There are no significant early differences in the processing of congruent and incongruent audiovisual action sequences. The earliest difference between congruent and incongruent audiovisual stimuli occurs between 240 and 280 ms after stimulus onset in the left temporal region. Between 340 and 420 ms, semantic congruence modulates responses in central and right frontal areas. Late differences (after 460 ms) occur bilaterally in frontal areas.2) Source localisation (dipole modelling and LORETA) reveals that an extended network encompassing inferior frontal, temporal, parasaggital, and superior parietal sites are simultaneously active between 180 and 420 ms to process auditory–visual action sequences. Early activation (before 120 ms) can be explained by activity in mainly sensory cortices. . The simultaneous activation of an extended network between 180 and 420 ms is consistent with models that posit parallel processing of complex action sequences in frontal, temporal and parietal areas rather than models that postulate hierarchical processing in a sequence of brain regions.

Motion words selectively modulate direction discrimination sensitivity for threshold motion

Can speech selectively modulate the sensitivity of a sensory system so that, in the presence of a suitable linguistic context, the discrimination of certain perceptual features becomes more or less likely? In this study, participants heard upward or downward motion words followed by a single visual field of random dots moving upwards or downwards. The time interval between the onsets of the auditory and the visual stimuli was varied parametrically. Motion direction could be either discriminable (suprathreshold motion) or non-discriminable (threshold motion). Participants had to judge whether the dots were moving upward or downward. Results show a double dissociation between discrimination sensitivity (d′) and reaction times depending on whether vertical motion was above or at threshold. With suprathreshold motion, responses were faster for congruent directions of words and dots, but sensitivity was equal across conditions. With threshold motion, sensitivity was higher for congruent directions of words and dots, but responses were equally fast across conditions. The observed differences in sensitivity and response times were largest when the dots appeared 450 ms after word onset, that is, consistently with electrophysiology, at the time the up/down semantics of the word had become available. These data suggest that word meanings can alter the balance between signal and noise within the visual system and affect the perception of low-level sensory features.

Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing

Functional magnetic resonance imaging (MRI) showed that the superior temporal and occipital cortex are involved in multisensory integration. Probabilistic fiber tracking based on diffusion-weighted MRI suggests that multisensory processing is supported by white matter connections between auditory cortex and the temporal and occipital lobe. Here, we present a combined functional MRI and probabilistic fiber tracking study that reveals multisensory processing mechanisms that remained undetected by either technique alone. Ten healthy participants passively observed visually presented lip or body movements, heard speech or body action sounds, or were exposed to a combination of both. Bimodal stimulation engaged a temporal-occipital brain network including the multisensory superior temporal sulcus (msSTS), the lateral superior temporal gyrus (lSTG), and the extrastriate body area (EBA). A region-of-interest (ROI) analysis showed multisensory interactions (e.g., subadditive responses to bimodal compared to unimodal stimuli) in the msSTS, the lSTG, and the EBA region. Moreover, sounds elicited responses in the medial occipital cortex. Probabilistic tracking revealed white matter tracts between the auditory cortex and the medial occipital cortex, the inferior occipital cortex (IOC), and the superior temporal sulcus (STS). However, STS terminations of auditory cortex tracts showed limited overlap with the msSTS region. Instead, msSTS was connected to primary sensory regions via intermediate nodes in the temporal and occipital cortex. Similarly, the lSTG and EBA regions showed limited direct white matter connections but instead were connected via intermediate nodes. Our results suggest that multisensory processing in the STS is mediated by separate brain areas that form a distinct network in the lateral temporal and inferior occipital cortex.

Reference repulsion in the categorical perception of biological motion

Perceiving biological motion is important for understanding the intentions and future actions of others. Perceiving an approaching person's behavior may be particularly important, because such behavior often precedes social interaction. To this end, the visual system may devote extra resources for perceiving an oncoming person's heading. If this were true, humans should show increased sensitivity for perceiving approaching headings, and as a result, a repulsive perceptual effect around the categorical boundary of leftward/rightward motion. We tested these predictions and found evidence for both. First, observers were especially sensitive to the heading of an approaching person; variability in estimates of a person's heading decreased near the category boundary of leftward/rightward motion. Second, we found a repulsion effect around the category boundary; a person walking approximately toward the observer was perceived as being repelled away from straight ahead. This repulsive effect was greatly exaggerated for perception of a very briefly presented person or perception of a chaotic crowd, suggesting that repulsion may protect against categorical errors when sensory noise is high. The repulsion effect with a crowd required integration of local motion and human form, suggesting an origin in high-level stages of visual processing. Similar repulsive effects may underlie categorical perception with other social features. Overall, our results show that a person's direction of walking is categorically perceived, with improved sensitivity at the category boundary and a concomitant repulsion effect.