Spatiotemporal Processing in Crossmodal Interactions for Perception of the External World: A Review

Unconscious multisensory integration: behavioral and neural evidence from subliminal stimuli

Introduction

The prevailing theories of consciousness consider the integration of different sensory stimuli as a key component for this phenomenon to rise on the brain level. Despite many theories and models have been proposed for multisensory integration between supraliminal stimuli (e.g., the optimal integration model), we do not know if multisensory integration occurs also for subliminal stimuli and what psychophysical mechanisms it follows.

Methods

To investigate this, subjects were exposed to visual (Virtual Reality) and/or haptic stimuli (Electro-Cutaneous Stimulation) above or below their perceptual threshold. They had to discriminate, in a two-Alternative Forced Choice Task, the intensity of unimodal and/or bimodal stimuli. They were then asked to discriminate the sensory modality while recording their EEG responses.

Results

We found evidence of multisensory integration for supraliminal condition, following the classical optimal model. Importantly, even for subliminal trials participant's performances in the bimodal condition were significantly more accurate when discriminating the intensity of the stimulation. Moreover, significant differences emerged between unimodal and bimodal activity templates in parieto-temporal areas known for their integrative role.

Discussion

These converging evidences - even if preliminary and needing confirmation from the collection of further data - suggest that subliminal multimodal stimuli can be integrated, thus filling a meaningful gap in the debate about the relationship between consciousness and multisensory integration.

Hearing water temperature: characterizing the development of nuanced perception of sound sources

Without conscious thought, listeners link events in the world to sounds they hear. We study one surprising example: Adults can judge the temperature of water simply from hearing it being poured. We test development of the ability to hear water temperature, with the goal of informing developmental theories regarding the origins and cognitive bases of nuanced sound source judgments. We first confirmed that adults accurately distinguished the sounds of hot and cold water (pre-registered Exps. 1, 2; total N = 384), even though many were unaware or uncertain of this ability. By contrast, children showed protracted development of this skill over the course of middle childhood (Exps. 2, 3; total N = 178). In spite of accurately identifying other sounds and hot/cold images, older children (7-11 years) but not younger children (3-6 years) reliably distinguished the sounds of hot and cold water. Accuracy increased with age; 11 year old's performance was similar to adults'. Adults also showed individual differences in accuracy that were predicted by their amount of prior relevant experience (Exp. 1). Experience may similarly play a role in children's performance; differences in auditory sensitivity and multimodal integration may also contribute to young children's failures. The ability to hear water temperature develops slowly over childhood, such that nuanced auditory information that is easily and quickly accessible to adults is not available to guide young children's behavior. Adults can make nuanced judgments from sound, including accurately judging the temperature of water from the sound of it being poured. Children showed protracted development of this skill over the course of middle childhood, such that 7-11 year-olds reliably succeeded while 3-6 year-olds performed at chance. Developmental changes may be due to experience (adults with greater relevant experience showed higher accuracy) and development of multimodal integration and auditory sensitivity. Young children may not detect subtle auditory information that adults easily perceive. This article is protected by copyright. All rights reserved.

Alterations of Prefrontal-Posterior Information Processing Patterns in Autism Spectrum Disorders

Autism spectrum disorder (ASD) is a heterogeneous disorder characterized by different levels of repetitive and stereotypic behavior as well as deficits in social interaction and communication. In this current study, we explored the changes in cerebral neural activities in ASD. The purpose of this study is to investigate whether there exists a dysfunction of interactive information processing between the prefrontal cortex and posterior brain regions in ASD. We investigated the atypical connectivity and information flow between the prefrontal cortex and posterior brain regions in ASD utilizing the entropy connectivity (a kind of directional connectivity) method. Eighty-nine patients with ASD and 94 typical developing (TD) teenagers participated in this study. Two-sample t-tests revealed weakened interactive entropy connectivity between the prefrontal cortex and posterior brain regions. This result indicates that there exists interactive prefrontal-posterior underconnectivity in ASD, and this disorder might lead to less prior knowledge being used and updated. Our proposals highlighted that aforementioned atypical change might accelerate the deoptimization of brain networks in ASD.

Cross-modal motion aftereffects transfer between vision and touch in early deaf adults

Previous research on early deafness has primarily focused on the behavioral and neural changes in the intact visual and tactile modalities. However, how early deafness changes the interplay of these two modalities is not well understood. In the current study, we investigated the effect of auditory deprivation on visuo-tactile interaction by measuring the cross-modal motion aftereffect. Consistent with previous findings, motion aftereffect transferred between vision and touch in a bidirectional manner in hearing participants. However, for deaf participants, the cross-modal transfer occurred only in the tactile-to-visual direction but not in the visual-to-tactile direction. This unidirectional cross-modal motion aftereffect found in the deaf participants could not be explained by unisensory motion aftereffect or discrimination threshold. The results suggest a reduced visual influence on tactile motion perception in early deaf individuals.

Effects of Auditory Patterns on Judged Displacements of an Occluded Moving Object

Using displays in which a moving disk disappeared behind an occluder, we examined whether an accompanying auditory rhythm influenced the perceived displacement of the disk during occlusion. We manipulated a baseline rhythm, comprising a relatively fast alternation of equal sound and pause durations. We had two different manipulations to create auditory sequences with a slower rhythm: either the pause durations or the sound durations were increased. In the trial, a disk moved at a constant speed, and at a certain point moved behind an occluder during which an auditory rhythm was played. Participants were instructed to track the occluded disk, and judge the expected position of the disk at the moment that the auditory rhythm ended by touching the judged position on a touch screen. We investigated the influence of the auditory rhythm, i.e., ratio of sound to pause duration, and the influence of auditory density, i.e., the number of sound onsets per time unit, on the judged distance. The results showed that the temporal characteristics affected the spatial judgments. Overall, we found that in the current paradigm relatively slow rhythms led to shorter judged distance as compared to relatively fast rhythms for both pause and sound variations. There was no main effect of auditory density on the judged distance of an expected visual event. That is, whereas the speed of the auditory rhythm appears crucial, the number of sound onsets per time unit as such, i.e., the auditory density, appears a much weaker factor.

On the effects of multimodal information integration in multitasking

There have recently been considerable advances in our understanding of the neuronal mechanisms underlying multitasking, but the role of multimodal integration for this faculty has remained rather unclear. We examined this issue by comparing different modality combinations in a multitasking (stop-change) paradigm. In-depth neurophysiological analyses of event-related potentials (ERPs) were conducted to complement the obtained behavioral data. Specifically, we applied signal decomposition using second order blind identification (SOBI) to the multi-subject ERP data and source localization. We found that both general multimodal information integration and modality-specific aspects (potentially related to task difficulty) modulate behavioral performance and associated neurophysiological correlates. Simultaneous multimodal input generally increased early attentional processing of visual stimuli (i.e. P1 and N1 amplitudes) as well as measures of cognitive effort and conflict (i.e. central P3 amplitudes). Yet, tactile-visual input caused larger impairments in multitasking than audio-visual input. General aspects of multimodal information integration modulated the activity in the premotor cortex (BA 6) as well as different visual association areas concerned with the integration of visual information with input from other modalities (BA 19, BA 21, BA 37). On top of this, differences in the specific combination of modalities also affected performance and measures of conflict/effort originating in prefrontal regions (BA 6).

Neural mechanisms underlying sound-induced visual motion perception: An fMRI study

Studies of crossmodal interactions in motion perception have reported activation in several brain areas, including those related to motion processing and/or sensory association, in response to multimodal (e.g., visual and auditory) stimuli that were both in motion. Recent studies have demonstrated that sounds can trigger illusory visual apparent motion to static visual stimuli (sound-induced visual motion: SIVM): A visual stimulus blinking at a fixed location is perceived to be moving laterally when an alternating left-right sound is also present. Here, we investigated brain activity related to the perception of SIVM using a 7T functional magnetic resonance imaging technique. Specifically, we focused on the patterns of neural activities in SIVM and visually induced visual apparent motion (VIVM). We observed shared activations in the middle occipital area (V5/hMT), which is thought to be involved in visual motion processing, for SIVM and VIVM. Moreover, as compared to VIVM, SIVM resulted in greater activation in the superior temporal area and dominant functional connectivity between the V5/hMT area and the areas related to auditory and crossmodal motion processing. These findings indicate that similar but partially different neural mechanisms could be involved in auditory-induced and visually-induced motion perception, and neural signals in auditory, visual, and, crossmodal motion processing areas closely and directly interact in the perception of SIVM.

Auditory Rhythms Influence Judged Time to Contact of an Occluded Moving Object

We studied the expected moment of reappearance of a moving object after it disappeared from sight. In particular, we investigated whether auditory rhythms influence time to contact (TTC) judgments. Using displays in which a moving disk disappears behind an occluder, we examined whether an accompanying auditory rhythm influences the expected TTC of an occluded moving object. We manipulated a baseline auditory rhythm — consisting of equal sound and pause durations — in two ways: either the pause durations or the sound durations were increased to create slower rhythms. Participants had to press a button at the moment they expected the disk to reappear. Variations in pause duration (Experiments 1 and 2) affected expected TTC, in contrast to variations in sound duration (Experiment 3). These results show that auditory rhythms affect expected reappearance of an occluded moving object. Second, these results suggest that temporal auditory grouping is an important factor in TTC.