Identifying and quantifying multisensory integration: a tutorial review

Psychological dissociation and temporal integration/segregation across the senses: An experimental study

There are no studies that have experimentally tested how temporal integration/segregation of sensory inputs might be linked to the emergence of dissociative experiences and alterations of emotional functioning. Thirty-six participants completed 3 sensory integration tasks. Psychometric thresholds were estimated as indexes of temporal integration/segregation processes. We collected self-report measures of pre-task trait levels of dissociation, as well as pre- post-task changes in both dissociation and emotionality. An independent sample of 21 subjects completed a control experiment administering the Attention Network Test. Results showed: (i) a significant increase of dissociative experiences after the completion of sensory integration tasks, but not after the ANT task; (ii) that subjective thresholds predicted the emergence of dissociative states; (iii) temporal integration efforts affected positive emotionality, which was explained by the extent of task-dependent dissociative states. The present findings reveal that dissociation could be understood in terms of an imbalance between "hyper-segregation" and "hyper-integration" processes.

Multisensory integration of speech and gestures in a naturalistic paradigm

Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power-an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.

Unsupervised separation of nonlinearly mixed event-related potentials using manifold clustering and non-negative matrix factorization

Event-related potentials (ERPs) can quantify brain responses to reveal the neural mechanisms of sensory perception. However, ERPs often reflect nonlinear mixture responses to multiple sources of sensory stimuli, and an accurate separation of the response to each stimulus remains a challenge. This study aimed to separate the ERP into nonlinearly mixed source components specific to individual stimuli. We developed an unsupervised learning method based on clustering of manifold structures of mixture signals combined with channel optimization for signal source reconstruction using non-negative matrix factorization (NMF). Specifically, we first implemented manifold learning based on Local Tangent Space Alignment (LTSA) to extract the spatial manifold structure of multi-resolution sub-signals separated via wavelet packet transform. We then used fuzzy entropy to extract the dynamical process of the manifold structures and performed a k-means clustering to separate different sources. Lastly, we used NMF to obtain the optimal contributions of multiple channels to ensure accurate source reconstructions. We evaluated our developed approach using a simulated ERP dataset with known ground truth of two components of ERP mixture signals. Our results show that the correlation coefficient between the reconstructed source signal and the true source signal was 92.8 % and that the separation accuracy in ERP amplitude was 91.6 %. The results show that our unsupervised separation approach can accurately separate ERP signals from nonlinear mixture source components. The outcomes provide a promising way to isolate brain responses to multiple stimulus sources during multisensory perception.

Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning

Social communication draws on several cognitive functions such as perception, emotion recognition and attention. The association of audio-visual information is essential to the processing of species-specific communication signals. In this study, we use functional magnetic resonance imaging in order to identify the subcortical areas involved in the cross-modal association of visual and auditory information based on their common social meaning. We identified three subcortical regions involved in audio-visual processing of species-specific communicative signals: the dorsolateral amygdala, the claustrum and the pulvinar. These regions responded to visual, auditory congruent and audio-visual stimulations. However, none of them was significantly activated when the auditory stimuli were semantically incongruent with the visual context, thus showing an influence of visual context on auditory processing. For example, positive vocalization (coos) activated the three subcortical regions when presented in the context of positive facial expression (lipsmacks) but not when presented in the context of negative facial expression (aggressive faces). In addition, the medial pulvinar and the amygdala presented multisensory integration such that audiovisual stimuli resulted in activations that were significantly higher than those observed for the highest unimodal response. Last, the pulvinar responded in a task-dependent manner, along a specific spatial sensory gradient. We propose that the dorsolateral amygdala, the claustrum and the pulvinar belong to a multisensory network that modulates the perception of visual socioemotional information and vocalizations as a function of the relevance of the stimuli in the social context. SIGNIFICANCE STATEMENT: Understanding and correctly associating socioemotional information across sensory modalities, such that happy faces predict laughter and escape scenes predict screams, is essential when living in complex social groups. With the use of functional magnetic imaging in the awake macaque, we identify three subcortical structures-dorsolateral amygdala, claustrum and pulvinar-that only respond to auditory information that matches the ongoing visual socioemotional context, such as hearing positively valenced coo calls and seeing positively valenced mutual grooming monkeys. We additionally describe task-dependent activations in the pulvinar, organizing along a specific spatial sensory gradient, supporting its role as a network regulator.

Smooth pursuit inhibition reveals audiovisual enhancement of fast movement control

The sudden onset of a visual object or event elicits an inhibition of eye movements at latencies approaching the minimum delay of visuomotor conductance in the brain. Typically, information presented via multiple sensory modalities, such as sound and vision, evokes stronger and more robust responses than unisensory information. Whether and how multisensory information affects ultra-short latency oculomotor inhibition is unknown. In two experiments, we investigate smooth pursuit and saccadic inhibition in response to multisensory distractors. Observers tracked a horizontally moving dot and were interrupted by an unpredictable visual, auditory, or audiovisual distractor. Distractors elicited a transient inhibition of pursuit eye velocity and catch-up saccade rate within ∼100 ms of their onset. Audiovisual distractors evoked stronger oculomotor inhibition than visual- or auditory-only distractors, indicating multisensory response enhancement. Multisensory response enhancement magnitudes were equal to the linear sum of responses to component stimuli. These results demonstrate that multisensory information affects eye movements even at ultra-short latencies, establishing a lower time boundary for multisensory-guided behavior. We conclude that oculomotor circuits must have privileged access to sensory information from multiple modalities, presumably via a fast, subcortical pathway.

Separation of bimodal fMRI responses in mouse somatosensory areas into V1 and non-V1 contributions

Multisensory integration is necessary for the animal to survive in the real world. While conventional methods have been extensively used to investigate the multisensory integration process in various brain areas, its long-range interactions remain less explored. In this study, our goal was to investigate interactions between visual and somatosensory networks on a whole-brain scale using 15.2-T BOLD fMRI. We compared unimodal to bimodal BOLD fMRI responses and dissected potential cross-modal pathways with silencing of primary visual cortex (V1) by optogenetic stimulation of local GABAergic neurons. Our data showed that the influence of visual stimulus on whisker activity is higher than the influence of whisker stimulus on visual activity. Optogenetic silencing of V1 revealed that visual information is conveyed to whisker processing via both V1 and non-V1 pathways. The first-order ventral posteromedial thalamic nucleus (VPM) was functionally affected by non-V1 sources, while the higher-order posterior medial thalamic nucleus (POm) was predominantly modulated by V1 but not non-V1 inputs. The primary somatosensory barrel field (S1BF) was influenced by both V1 and non-V1 inputs. These observations provide valuable insights for into the integration of whisker and visual sensory information.

Planning lane changes using advance visual and haptic information

Taking a motor planning perspective, this study investigates whether haptic force cues displayed on the steering wheel are more effective than visual cues in signaling the direction of an upcoming lane change. Licensed drivers drove in a fixed-base driving simulator equipped with an active steering system for realistic force feedback. They were instructed to make lane changes upon registering a directional cue. Cues were delivered according to the movement precuing technique employing a pair of precues and imperative cues which could be either visual, haptic, or crossmodal (a visual precue with a haptic imperative cue, and vice versa). The main dependent variable was response time. Additional analyses were conducted on steering wheel angle profiles and the rate of initial steering errors. Conditions with a haptic imperative cue produced considerably faster responses than conditions with a visual imperative cue, irrespective of the precue modality. Valid and invalid precues produced the typical gains and costs, with one exception. There appeared to be little cost in response time or initial steering errors associated with invalid cueing when both cues were haptic. The results are consistent with the hypothesis that imperative haptic cues facilitate action selection while visual stimuli require additional time-consuming cognitive processing.

Improved speech intelligibility in the presence of congruent vibrotactile speech input

Vibrotactile stimulation is believed to enhance auditory speech perception, offering potential benefits for cochlear implant (CI) users who may utilize compensatory sensory strategies. Our study advances previous research by directly comparing tactile speech intelligibility enhancements in normal-hearing (NH) and CI participants, using the same paradigm. Moreover, we assessed tactile enhancement considering stimulus non-specific, excitatory effects through an incongruent audio-tactile control condition that did not contain any speech-relevant information. In addition to this incongruent audio-tactile condition, we presented sentences in an auditory only and a congruent audio-tactile condition, with the congruent tactile stimulus providing low-frequency envelope information via a vibrating probe on the index fingertip. The study involved 23 NH listeners and 14 CI users. In both groups, significant tactile enhancements were observed for congruent tactile stimuli (5.3% for NH and 5.4% for CI participants), but not for incongruent tactile stimulation. These findings replicate previously observed tactile enhancement effects. Juxtaposing our study with previous research, the informational content of the tactile stimulus emerges as a modulator of intelligibility: Generally, congruent stimuli enhanced, non-matching tactile stimuli reduced, and neutral stimuli did not change test outcomes. We conclude that the temporal cues provided by congruent vibrotactile stimuli may aid in parsing continuous speech signals into syllables and words, consequently leading to the observed improvements in intelligibility.

A Comparison of Listening Skills of Autistic and Non-Autistic Youth While Using and Not Using Remote Microphone Systems

OBJECTIVES

The purposes of this study were to compare (a) listening-in-noise (accuracy and effort) and (b) remote microphone (RM) system benefits between autistic and non-autistic youth.

DESIGN

Groups of autistic and non-autistic youth that were matched on chronological age and biological sex completed listening-in-noise testing when wearing and not wearing an RM system. Listening-in-noise accuracy and listening effort were evaluated simultaneously using a dual-task paradigm for stimuli varying in type (syllables, words, sentences, and passages). Several putative moderators of RM system effects on outcomes of interest were also evaluated.

RESULTS

Autistic youth outperformed non-autistic youth in some conditions on listening-in-noise accuracy; listening effort between the two groups was not significantly different. RM system use resulted in listening-in-noise accuracy improvements that were nonsignificantly different across groups. Benefits of listening-in-noise accuracy were all large in magnitude. RM system use did not have an effect on listening effort for either group. None of the putative moderators yielded effects of the RM system on listening-in-noise accuracy or effort for non-autistic youth that were significant and interpretable, indicating that RM system benefits did not vary according to any of the participant characteristics assessed.

CONCLUSIONS

Contrary to expectations, autistic youth did not demonstrate listening-in-noise deficits compared to non-autistic youth. Both autistic and non-autistic youth appear to experience RM system benefits marked by large gains in listening-in-noise performance. Thus, the use of this technology in educational and other noisy settings where speech perception needs enhancement might be beneficial for both groups of children.

Audiotactile Stimulation Can Improve Syllable Discrimination through Multisensory Integration in the Theta Frequency Band

Syllables are an essential building block of speech. We recently showed that tactile stimuli linked to the perceptual centers of syllables in continuous speech can improve speech comprehension. The rate of syllables lies in the theta frequency range, between 4 and 8 Hz, and the behavioral effect appears linked to multisensory integration in this frequency band. Because this neural activity may be oscillatory, we hypothesized that a behavioral effect may also occur not only while but also after this activity has been evoked or entrained through vibrotactile pulses. Here, we show that audiotactile integration regarding the perception of single syllables, both on the neural and on the behavioral level, is consistent with this hypothesis. We first stimulated participants with a series of vibrotactile pulses and then presented them with a syllable in background noise. We show that, at a delay of 200 msec after the last vibrotactile pulse, audiotactile integration still occurred in the theta band and syllable discrimination was enhanced. Moreover, the dependence of both the neural multisensory integration as well as of the behavioral discrimination on the delay of the audio signal with respect to the last tactile pulse was consistent with a damped oscillation. In addition, the multisensory gain is correlated with the syllable discrimination score. Our results therefore evidence the role of the theta band in audiotactile integration and provide evidence that these effects may involve oscillatory activity that still persists after the tactile stimulation.

Deafness induces complete crossmodal plasticity in a belt region of dorsal auditory cortex

Many neural areas, where patterned activity is lost following deafness, have the capacity to become activated by the remaining sensory systems. This crossmodal plasticity can be measured at perceptual/behavioural as well as physiological levels. The dorsal zone (DZ) of auditory cortex of deaf cats is involved in supranormal visual motion detection, but its physiological level of crossmodal reorganisation is not well understood. The present study of early-deaf DZ (and hearing controls) used multiple single-channel recording methods to examine neuronal responses to visual, auditory, somatosensory and combined stimulation. In early-deaf DZ, no auditory activation was observed, but 100% of the neurons were responsive to visual cues of which 21% were also influenced by somatosensory stimulation. Visual and somatosensory responses were not anatomically organised as they are in hearing cats, and fewer multisensory neurons were present in the deaf condition. These crossmodal physiological results closely correspond with and support the perceptual/behavioural enhancements that occur following hearing loss.

Effects of continuous tactile stimulation on auditory-evoked cortical responses depend on the audio-tactile phase

Auditory perception can benefit from stimuli in non-auditory sensory modalities, as for example in lip-reading. Compared with such visual influences, tactile influences are still poorly understood. It has been shown that single tactile pulses can enhance the perception of auditory stimuli depending on their relative timing, but whether and how such brief auditory enhancements can be stretched in time with more sustained, phase-specific periodic tactile stimulation is still unclear. To address this question, we presented tactile stimulation that fluctuated coherently and continuously at 4Hz with an auditory noise (either in-phase or anti-phase) and assessed its effect on the cortical processing and perception of an auditory signal embedded in that noise. Scalp-electroencephalography recordings revealed an enhancing effect of in-phase tactile stimulation on cortical responses phase-locked to the noise and a suppressive effect of anti-phase tactile stimulation on responses evoked by the auditory signal. Although these effects appeared to follow well-known principles of multisensory integration of discrete audio-tactile events, they were not accompanied by corresponding effects on behavioral measures of auditory signal perception. Our results indicate that continuous periodic tactile stimulation can enhance cortical processing of acoustically-induced fluctuations and mask cortical responses to an ongoing auditory signal. They further suggest that such sustained cortical effects can be insufficient for inducing sustained bottom-up auditory benefits.

Neural correlates of multisensory integration in the human brain: an ALE meta-analysis

Previous fMRI research identified superior temporal sulcus as central integration area for audiovisual stimuli. However, less is known about a general multisensory integration network across senses. Therefore, we conducted activation likelihood estimation meta-analysis with multiple sensory modalities to identify a common brain network. We included 49 studies covering all Aristotelian senses i.e., auditory, visual, tactile, gustatory, and olfactory stimuli. Analysis revealed significant activation in bilateral superior temporal gyrus, middle temporal gyrus, thalamus, right insula, and left inferior frontal gyrus. We assume these regions to be part of a general multisensory integration network comprising different functional roles. Here, thalamus operate as first subcortical relay projecting sensory information to higher cortical integration centers in superior temporal gyrus/sulcus while conflict-processing brain regions as insula and inferior frontal gyrus facilitate integration of incongruent information. We additionally performed meta-analytic connectivity modelling and found each brain region showed co-activations within the identified multisensory integration network. Therefore, by including multiple sensory modalities in our meta-analysis the results may provide evidence for a common brain network that supports different functional roles for multisensory integration.

Effect of anticipatory multisensory integration on sensory-motor performance

Multisensory integration (MSI) is a phenomenon that occurs in sensory areas after the presentation of multimodal stimuli. Nowadays, little is known about the anticipatory top-down processes taking place in the preparation stage of processing before the stimulus onset. Considering that the top-down modulation of modality-specific inputs might affect the MSI process, this study attempts to understand whether the direct modulation of the MSI process, beyond the well-known sensory effects, may lead to additional changes in multisensory processing also in non-sensory areas (i.e., those related to task preparation and anticipation). To this aim, event-related potentials (ERPs) were analyzed both before and after auditory and visual unisensory and multisensory stimuli during a discriminative response task (Go/No-go type). Results showed that MSI did not affect motor preparation in premotor areas, while cognitive preparation in the prefrontal cortex was increased and correlated with response accuracy. Early post-stimulus ERP activities were also affected by MSI and correlated with response time. Collectively, the present results point to the plasticity accommodating nature of the MSI processes, which are not limited to perception and extend to anticipatory cognitive preparation for task execution. Further, the enhanced cognitive control emerging during MSI is discussed in the context of Bayesian accounts of augmented predictive processing related to increased perceptual uncertainty.

Health Implications of Virtual Architecture: An Interdisciplinary Exploration of the Transferability of Findings from Neuroarchitecture

Virtual architecture has been increasingly relied on to evaluate the health impacts of physical architecture. In this health research, exposure to virtual architecture has been used as a proxy for exposure to physical architecture. Despite the growing body of research on the health implications of physical architecture, there is a paucity of research examining the long-term health impacts of prolonged exposure to virtual architecture. In response, this paper considers: what can proxy studies, which use virtual architecture to assess the physiological response to physical architecture, tell us about the impact of extended exposure to virtual architecture on human health? The paper goes on to suggest that the applicability of these findings to virtual architecture may be limited by certain confounding variables when virtual architecture is experienced for a prolonged period of time. This paper explores the potential impact of two of these confounding variables: multisensory integration and gravitational perception. This paper advises that these confounding variables are unique to extended virtual architecture exposure and may not be captured by proxy studies that aim to capture the impact of physical architecture on human health through acute exposure to virtual architecture. While proxy studies may be suitable for measuring some aspects of the impact of both physical and virtual architecture on human health, this paper argues that they may be insufficient to fully capture the unintended consequences of extended exposure to virtual architecture on human health. Therefore, in the face of the increasing use of virtual architectural environments, the author calls for the establishment of a subfield of neuroarchitectural health research that empirically examines the physiological impacts of extended exposure to virtual architecture in its own right.

Working memory load modulates the processing of audiovisual distractors: A behavioral and event-related potentials study

The interplay between different modalities can help to perceive stimuli more effectively. However, very few studies have focused on how multisensory distractors affect task performance. By adopting behavioral and event-related potentials (ERPs) techniques, the present study examined whether multisensory audiovisual distractors could attract attention more effectively than unisensory distractors. Moreover, we explored whether such a process was modulated by working memory load. Across three experiments, n-back tasks (1-back and 2-back) were adopted with peripheral auditory, visual, or audiovisual distractors. Visual and auditory distractors were white discs and pure tones (Experiments 1 and 2), pictures and sounds of animals (Experiment 3), respectively. Behavioral results in Experiment 1 showed a significant interference effect under high working memory load but not under low load condition. The responses to central letters with audiovisual distractors were significantly slower than those to letters without distractors, while no significant difference was found between unisensory distractor and without distractor conditions. Similarly, ERP results in Experiments 2 and 3 showed that there existed an integration only under high load condition. That is, an early integration for simple audiovisual distractors (240-340 ms) and a late integration for complex audiovisual distractors (440-600 ms). These findings suggest that multisensory distractors can be integrated and effectively attract attention away from the main task, i.e., interference effect. Moreover, this effect is pronounced only under high working memory load condition.

Aberrant neural correlates of multisensory processing of audiovisual social cues related to social anxiety: An electrophysiological study

BACKGROUND

Social anxiety disorder (SAD) is characterized by abnormal fear to social cues. Although unisensory processing to social stimuli associated with social anxiety (SA) has been well described, how multisensory processing relates to SA is still open to clarification. Using electroencephalography (EEG) measurement, we investigated the neural correlates of multisensory processing and related temporal dynamics in social anxiety disorder (SAD).

METHODS

Twenty-five SAD participants and 23 healthy control (HC) participants were presented with angry and neutral faces, voices and their combinations with congruent emotions and they completed an emotional categorization task.

RESULTS

We found that face-voice combinations facilitated auditory processing in multiple stages indicated by the acceleration of auditory N1 latency, attenuation of auditory N1 and P250 amplitudes, and decrease of theta power. In addition, bimodal inputs elicited cross-modal integrative activity which is indicated by the enhancement of visual P1, N170, and P3/LPP amplitudes and superadditive response of P1 and P3/LPP. More importantly, excessively greater integrative activity (at P3/LPP amplitude) was found in SAD participants, and this abnormal integrative activity in both early and late temporal stages was related to the larger interpretation bias of miscategorizing neutral face-voice combinations as angry.

CONCLUSION

The study revealed that neural correlates of multisensory processing was aberrant in SAD and it was related to the interpretation bias to multimodal social cues in multiple processing stages. Our findings suggest that deficit in multisensory processing might be an important factor in the psychopathology of SA.

Multisensory integration in humans with spinal cord injury

Although multisensory integration (MSI) has been extensively studied, the underlying mechanisms remain a topic of ongoing debate. Here we investigate these mechanisms by comparing MSI in healthy controls to a clinical population with spinal cord injury (SCI). Deafferentation following SCI induces sensorimotor impairment, which may alter the ability to synthesize cross-modal information. We applied mathematical and computational modeling to reaction time data recorded in response to temporally congruent cross-modal stimuli. We found that MSI in both SCI and healthy controls is best explained by cross-modal perceptual competition, highlighting a common competition mechanism. Relative to controls, MSI impairments in SCI participants were better explained by reduced stimulus salience leading to increased cross-modal competition. By combining traditional analyses with model-based approaches, we examine how MSI is realized during normal function, and how it is compromised in a clinical population. Our findings support future investigations identifying and rehabilitating MSI deficits in clinical disorders.

Sensorimotor processing is dependent on observed speed during the observation of hand-hand and hand-object interactions

Observing a physical interaction between individuals (e.g., observing friends shaking hands) or between an object and an individual (e.g., observing a teammate striking or being struck with a ball) can lead to somatosensory activation in the observer. However, it is not known whether the speed of the observed interaction modulates such somatosensory activation (e.g., observing a teammate being struck with a slow vs. a fast-moving ball). In three experiments, participants observed a hand or object interact with another hand or object, all presented with a slow- or fast-moving effector. To probe sensorimotor processes during observation, participants were asked to react to an auditory beep (i.e., response time [RT] task) at the moment of observed contact. If observed contact led to increased somatosensory activation, RTs would decrease due to statistical and/ or intersensory facilitation. In all three experiments, RTs were lower when observing fast compared to slow motion stimuli, regardless of the moving (i.e., hand or ball) and target stimulus (i.e., hand or leaf). Further, when only an object (i.e., leaf) was the target, RTs did not differ between the moving hand and moving ball condition. In contrast, when an object (i.e., ball) was used as the moving stimulus, the magnitude of the speed effect (i.e., fast - slow RT difference) was significantly larger when the ball contacted a hand as compared to a leaf. Overall, these results provide novel evidence for a relationship between the observed kinematics of an object-human interaction and the sensorimotor processing in the observer.

A comparison of multisensory features of two auditory cortical areas: primary (A1) and higher-order dorsal zone (DZ)

From myriads of ongoing stimuli, the brain creates a fused percept of the environment. This process, which culminates in perceptual binding, is presumed to occur through the operations of multisensory neurons that occur throughout the brain. However, because different brain areas receive different inputs and have different cytoarchitechtonics, it would be expected that local multisensory features would also vary across regions. The present study investigated that hypothesis using multiple single-unit recordings from anesthetized cats in response to controlled, electronically-generated separate and combined auditory, visual, and somatosensory stimulation. These results were used to compare the multisensory features of neurons in cat primary auditory cortex (A1) with those identified in the nearby higher-order auditory region, the Dorsal Zone (DZ). Both regions exhibited the same forms of multisensory neurons, albeit in different proportions. Multisensory neurons exhibiting excitatory or inhibitory properties occurred in similar proportions in both areas. Also, multisensory neurons in both areas expressed similar levels of multisensory integration. Because responses to auditory cues alone were so similar to those that included non-auditory stimuli, it is proposed that this effect represents a mechanism by which multisensory neurons subserve the process of perceptual binding.

An analytical method to separate modality-specific and nonspecific sensory components of event-related potentials

Several models have been developed to analyse cortical activity in response to salient events constituted by multiple sensory modalities. In particular, additive models compare event-related potentials (ERPs) in response to stimuli from two or more concomitant sensory modalities with the ERPs evoked by unimodal stimuli, in order to study sensory interactions. In this approach, components that are not specific to a sensory modality are commonly disregarded, although they likely carry information about stimulus expectation and evaluation, attentional orientation and other cognitive processes. In this study, we present an analytical method to assess the contribution of modality-specific and nonspecific components to the ERP. We developed an experimental setup that recorded ERPs in response to four stimulus types (visual, auditory, and two somatosensory modalities to test for stimulus specificity) in three different conditions (unimodal, bimodal and trimodal stimulation) and recorded the saliency of these stimuli relative to the sensory background. Stimuli were delivered in pairs, in order to study the effects of habituation. To this end, spatiotemporal features (peak amplitudes and latencies at different scalp locations) were analysed using linear mixed models. Results showed that saliency relative to the sensory background increased with the number of concomitant stimuli. We also observed that the spatiotemporal features of modality-specific components derived from this method likely reflect the amount and type of sensory input. Furthermore, the nonspecific component reflected habituation occurring for the second stimulus in the pair. In conclusion, this method provides an alternative to study neural mechanisms of responses to multisensory stimulation.

Are auditory cues special? Evidence from cross-modal distractor-induced blindness

A target that shares features with preceding distractor stimuli is less likely to be detected due to a distractor-driven activation of a negative attentional set. This transient impairment in perceiving the target (distractor-induced blindness/deafness) can be found within vision and audition. Recently, the phenomenon was observed in a cross-modal setting involving an auditory target and additional task-relevant visual information (cross-modal distractor-induced deafness). In the current study, consisting of three behavioral experiments, a visual target, indicated by an auditory cue, had to be detected despite the presence of visual distractors. Multiple distractors consistently led to reduced target detection if cue and target appeared in close temporal proximity, confirming cross-modal distractor-induced blindness. However, the effect on target detection was reduced compared to the effect of cross-modal distractor-induced deafness previously observed for reversed modalities. The physical features defining cue and target could not account for the diminished distractor effect in the current cross-modal task. Instead, this finding may be attributed to the auditory cue acting as an especially efficient release signal of the distractor-induced inhibition. Additionally, a multisensory enhancement of visual target detection by the concurrent auditory signal might have contributed to the reduced distractor effect.

The relationship between multisensory associative learning and multisensory integration

Integrating sensory information from multiple modalities leads to more precise and efficient perception and behaviour. The process of determining which sensory information should be perceptually bound is reliant on both low-level stimulus features, as well as multisensory associations learned throughout development based on the statistics of our environment. Here, we explored the relationship between multisensory associative learning and multisensory integration using encephalography (EEG) and behavioural measures. Sixty-one participants completed a three-phase study. First, participants were exposed to novel audiovisual shape-tone pairings with frequent and infrequent stimulus pairings and complete a target detection task. EEG recordings of the mismatch negativity (MMN) and P3 were calculated as neural indices of multisensory associative learning. Next, the same learned stimulus pairs were presented in audiovisual as well as unisensory auditory and visual modalities while both early (<120 ms) and late neural indices of multisensory integration were recorded. Finally, participants completed an analogous behavioural speeded-response task, with behavioural indices of multisensory gain calculated using the Race Model. Significant relationships were found in fronto-central and occipital areas between neural measures of associative learning and both early and late indices of multisensory integration in frontal and centro-parietal areas, respectively. Participants who showed stronger indices of associative learning also exhibited stronger indices of multisensory integration of the stimuli they learned to associate. Furthermore, a significant relationship was found between neural index of early multisensory integration and behavioural indices of multisensory gain. These results provide insight into the neural underpinnings of how higher-order processes such as associative learning guide multisensory integration.

Source Localization of Audiovisual Multisensory Neural Generators in Young Adults with Attention-Deficit/Hyperactivity Disorder

Attention-Deficit/Hyperactivity Disorder (ADHD) is a neurodevelopmental disorder that exhibits unique neurological and behavioural characteristics. Our previous work using event-related potentials demonstrated that adults with ADHD process audiovisual multisensory stimuli somewhat differently than neurotypical controls. This study utilised an audiovisual multisensory two-alternative forced-choice discrimination task. Continuous whole-head electroencephalography (EEG) was recorded. Source localization (sLORETA) software was utilised to determine differences in the contribution made by sources of neural generators pertinent to audiovisual multisensory processing in those with ADHD versus neurotypical controls. Source localization techniques elucidated that the controls had greater neural activity 164 ms post-stimulus onset when compared to the ADHD group, but only when responding to audiovisual stimuli. The source of the increased activity was found to be Brodmann Area 2, postcentral gyrus, right-hemispheric parietal lobe referenced to Montreal Neurological Institute (MNI) coordinates of X = 35, Y = −40, and Z = 70 (p < 0.05). No group differences were present during either of the unisensory conditions. Differences in the integration areas, particularly in the right-hemispheric parietal brain regions, were found in those with ADHD. These alterations may correspond to impaired attentional capabilities when presented with multiple simultaneous sensory inputs, as is the case during a multisensory condition.

The timecourse of multisensory speech processing in unilaterally stimulated cochlear implant users revealed by ERPs

•

Both normal-hearing (NH) and cochlear implant (CI) users show a clear benefit in multisensory speech processing.

•

Group differences in ERP topographies and cortical source activation suggest distinct audiovisual speech processing in CI users when compared to NH listeners.

•

Electrical neuroimaging, including topographic and ERP source analysis, provides a suitable tool to study the timecourse of multisensory speech processing in CI users.

A cochlear implant (CI) is an auditory prosthesis which can partially restore the auditory function in patients with severe to profound hearing loss. However, this bionic device provides only limited auditory information, and CI patients may compensate for this limitation by means of a stronger interaction between the auditory and visual system. To better understand the electrophysiological correlates of audiovisual speech perception, the present study used electroencephalography (EEG) and a redundant target paradigm. Postlingually deafened CI users and normal-hearing (NH) listeners were compared in auditory, visual and audiovisual speech conditions. The behavioural results revealed multisensory integration for both groups, as indicated by shortened response times for the audiovisual as compared to the two unisensory conditions. The analysis of the N1 and P2 event-related potentials (ERPs), including topographic and source analyses, confirmed a multisensory effect for both groups and showed a cortical auditory response which was modulated by the simultaneous processing of the visual stimulus. Nevertheless, the CI users in particular revealed a distinct pattern of N1 topography, pointing to a strong visual impact on auditory speech processing. Apart from these condition effects, the results revealed ERP differences between CI users and NH listeners, not only in N1/P2 ERP topographies, but also in the cortical source configuration. When compared to the NH listeners, the CI users showed an additional activation in the visual cortex at N1 latency, which was positively correlated with CI experience, and a delayed auditory-cortex activation with a reversed, rightward functional lateralisation. In sum, our behavioural and ERP findings demonstrate a clear audiovisual benefit for both groups, and a CI-specific alteration in cortical activation at N1 latency when auditory and visual input is combined. These cortical alterations may reflect a compensatory strategy to overcome the limited CI input, which allows the CI users to improve the lip-reading skills and to approximate the behavioural performance of NH listeners in audiovisual speech conditions. Our results are clinically relevant, as they highlight the importance of assessing the CI outcome not only in auditory-only, but also in audiovisual speech conditions.

Enhancement of speech-in-noise comprehension through vibrotactile stimulation at the syllabic rate

Syllables are important building blocks of speech. They occur at a rate between 4 and 8 Hz, corresponding to the theta frequency range of neural activity in the cerebral cortex. When listening to speech, the theta activity becomes aligned to the syllabic rhythm, presumably aiding in parsing a speech signal into distinct syllables. However, this neural activity cannot only be influenced by sound, but also by somatosensory information. Here, we show that the presentation of vibrotactile signals at the syllabic rate can enhance the comprehension of speech in background noise. We further provide evidence that this multisensory enhancement of speech comprehension reflects the multisensory integration of auditory and tactile information in the auditory cortex.

Speech unfolds over distinct temporal scales, in particular, those related to the rhythm of phonemes, syllables, and words. When a person listens to continuous speech, the syllabic rhythm is tracked by neural activity in the theta frequency range. The tracking plays a functional role in speech processing: Influencing the theta activity through transcranial current stimulation, for instance, can impact speech perception. The theta-band activity in the auditory cortex can also be modulated through the somatosensory system, but the effect on speech processing has remained unclear. Here, we show that vibrotactile feedback presented at the rate of syllables can modulate and, in fact, enhance the comprehension of a speech signal in background noise. The enhancement occurs when vibrotactile pulses occur at the perceptual center of the syllables, whereas a temporal delay between the vibrotactile signals and the speech stream can lead to a lower level of speech comprehension. We further investigate the neural mechanisms underlying the audiotactile integration through electroencephalographic (EEG) recordings. We find that the audiotactile stimulation modulates the neural response to the speech rhythm, as well as the neural response to the vibrotactile pulses. The modulations of these neural activities reflect the behavioral effects on speech comprehension. Moreover, we demonstrate that speech comprehension can be predicted by particular aspects of the neural responses. Our results evidence a role of vibrotactile information for speech processing and may have applications in future auditory prosthesis.

Whether attentional loads influence audiovisual integration depends on semantic associations

Neuronal studies have shown that selectively attending to a common object in one sensory modality results in facilitated processing of that object's representations in the ignored sensory modality. Thus, the audiovisual (AV) integration of common objects can be observed under modality-specific selective attention. However, little is known about whether this AV integration can also occur under increased attentional load conditions. Additionally, whether semantic associations between multisensory features of common objects modulate the influence of increased attentional loads on this cross-modal integration remains unknown. In the present study, participants completed an AV integration task (ignored auditory stimuli) under various attentional load conditions: no load, low load, and high load. The semantic associations between AV stimuli were composed of animal pictures presented concurrently with semantically congruent, semantically incongruent, or semantically unrelated auditory stimuli. Our results demonstrated that attentional loads did not disrupt the integration of semantically congruent AV stimuli but suppressed the potential alertness effects induced by incongruent or unrelated auditory stimuli under the condition of modality-specific selective attention. These findings highlight the critical role of semantic association between AV stimuli in modulating the effect of attentional loads on the AV integration of modality-specific selective attention.

Multisensory integration attenuates visually induced oculomotor inhibition of return

Inhibition of return (IOR) is a mechanism of the attention system involving bias toward novel stimuli and delayed generation of responses to targets at previously attended locations. According to the two-component theory, IOR consists of a perceptual component and an oculomotor component (oculomotor IOR [O-IOR]) depending on whether the eye movement system is activated. Previous studies have shown that multisensory integration weakens IOR when paying attention to both visual and auditory modalities. However, it remains unclear whether the O-IOR effect attenuated by multisensory integration also occurs when the oculomotor system is activated. Here, using two eye movement experiments, we investigated the effect of multisensory integration on O-IOR using the exogenous spatial cueing paradigm. In Experiment 1, we found a greater visual O-IOR effect compared with audiovisual and auditory O-IOR in divided modality attention. The relative multisensory response enhancement (rMRE) and violations of Miller's bound showed a greater magnitude of multisensory integration in the cued location compared with the uncued location. In Experiment 2, the magnitude of the audiovisual O-IOR effect was significantly less than that of the visual O-IOR in single visual modality selective attention. Implications for the effect of multisensory integration on O-IOR were discussed under conditions of oculomotor system activation, shedding new light on the two-component theory of IOR.

Less is more: Removing a modality of an expected olfactory-visual stimulation enhances brain activation

In recent years, multisensory integration of visual and olfactory stimuli has extensively been explored resulting in the identification of responsible brain areas. As the experimental designs of previous research often include alternating presentations of unimodal and bimodal stimuli, the conditions cannot be regarded as completely independent. This could lead to effects of an expected but surprisingly missing sensory modality. In our experiment, we used a common functional magnetic resonance imaging (fMRI) study design with alternating strong unimodal and bimodal olfactory-visual food stimuli, in addition to a slight overhang of the bimodal stimuli in an effort to examine the effects of removing a visual or olfactory congruent stimulus for older people (41-83 years). Our results suggest that the processing of olfactory and visual stimuli stays intact over a wide age-range and that the utilization of strong stimuli does not lead to superadditive multisensory integration in accordance with the principle of inverse effectiveness. However, our results demonstrate that the removal of a stimulus modality leads to an activation of additional brain areas. For example, when the visual stimulus modality is missing, the right posterior superior temporal gyrus shows higher activation, whereas the removal of the olfactory stimulus modality leads to higher activation in the amygdala/hippocampus and the postcentral gyrus. These brain areas are related to attention, memory, and the search of the missing stimulus. Consequently, careful attention must be paid to the design of a valid, multimodal sensory experiment while also controlling for cognitive expectancy effects that might confound multimodal results.

Functional localization of audiovisual speech using near infrared spectroscopy

Visual cues are especially vital for hearing impaired individuals such as cochlear implant (CI) users to understand speech in noise. Functional Near Infrared Spectroscopy (fNIRS) is a light-based imaging technology that is ideally suited for measuring the brain activity of CI users due to its compatibility with both the ferromagnetic and electrical components of these implants. In a preliminary step toward better elucidating the behavioral and neural correlates of audiovisual (AV) speech integration in CI users, we designed a speech-in-noise task and measured the extent to which 24 normal hearing individuals could integrate the audio of spoken monosyllabic words with the corresponding visual signals of a female speaker. In our behavioral task, we found that audiovisual pairings provided average improvements of 103% and 197% over auditory-alone listening conditions in -6 and -9 dB signal-to-noise ratios consisting of multi-talker background noise. In an fNIRS task using similar stimuli, we measured activity during auditory-only listening, visual-only lipreading, and AV listening conditions. We identified cortical activity in all three conditions over regions of middle and superior temporal cortex typically associated with speech processing and audiovisual integration. In addition, three channels active during the lipreading condition showed uncorrected correlations associated with behavioral measures of audiovisual gain as well as with the McGurk effect. Further work focusing primarily on the regions of interest identified in this study could test how AV speech integration may differ for CI users who rely on this mechanism for daily communication.

OpenSync: An opensource platform for synchronizing multiple measures in neuroscience experiments

BACKGROUND

The human mind is multimodal. Yet most behavioral studies rely on century-old measures such as task accuracy and latency. To create a better understanding of human behavior and brain functionality, we should introduce other measures and analyze behavior from various aspects. However, it is technically complex and costly to design and implement the experiments that record multiple measures. To address this issue, a platform that allows synchronizing multiple measures from human behavior is needed.

METHOD

This paper introduces an opensource platform named OpenSync, which can be used to synchronize multiple measures in neuroscience experiments. This platform helps to automatically integrate, synchronize and record physiological measures (e.g., electroencephalogram (EEG), galvanic skin response (GSR), eye-tracking, body motion, etc.), user input response (e.g., from mouse, keyboard, joystick, etc.), and task-related information (stimulus markers). In this paper, we explain the structure and details of OpenSync, provide two case studies in PsychoPy and Unity. Comparison with existing tools: Unlike proprietary systems (e.g., iMotions), OpenSync is free and it can be used inside any opensource experiment design software (e.g., PsychoPy, OpenSesame, Unity, etc., https://pypi.org/project/OpenSync/ and https://github.com/moeinrazavi/OpenSync_Unity).

RESULTS

Our experimental results show that the OpenSync platform is able to synchronize multiple measures with microsecond resolution.

MULTISENSORY RESPONSES IN A BELT REGION OF THE DORSAL AUDITORY CORTICAL PATHWAY

A basic function of the cerebral cortex is to receive and integrate information from different sensory modalities into a comprehensive percept of the environment. Neurons that demonstrate multisensory convergence occur across the necortex, but are especially prevalent in higher-order, association areas. However, a recent study of a cat higher-order auditory area, the dorsal zone (DZ) of auditory cortex, did not observe any multisensory features. Therefore, the goal of the present investigation was to address this conflict using recording and testing methodologies that are established for exposing and studying multisensory neuronal processing. Among the 482 neurons studied, we found that 76.6% were influenced by non-auditory stimuli. Of these neurons, 99% were affected by visual stimulation, but only 11% by somatosensory. Furthermore, a large proportion of the multisensory neurons showed integrated responses to multisensory stimulation, constituted a majority of the excitatory and inhibitory neurons encountered (as identified by the duration of their waveshape), and exhibited a distinct spatial distribution within DZ. These findings demonstrate that the dorsal zone of auditory cortex robustly exhibits multisensory properties and that the proportions of multisensory neurons encountered are consistent with those identified in other higher-order cortices.

Evaluation of Visual and Tactile Perception by Plain-Body Octopus (Callistoctopus aspilosomatis) of Prey-Like Objects

We investigated the characteristic features of perception in octopuses by examining multisensory information from an object simulating prey, which provided different visual and tactile stimuli. In experiments, we presented plain-body octopus with four kinds of models, namely, the Lifelike crab, the Embedded crab, the Translucent crab, and the Black cuboid. These models contain different amounts of visual and tactile information that a crab originally contains: the Lifelike crab resembles a crab both visually and tactilely, the Embedded crab resembles a crab visually but provides different tactile information, the Translucent crab provides tactile information of a crab but contains less visual information, and the Black cuboid lacks both visual and tactile information of a crab. Among these four models, octopuses contacted most with the Lifelike crab, which was similar to their behavior with a crab. Indeed, octopuses were fastest to contact the Lifelike crab and had the longest duration of contacting it among the four models. Octopuses contacted the Embedded crab more than the Translucent crab, both of which had contrasting visuo-tactile information compared to that of a crab. Quickness of octopuses to contact and duration of contact with the Embedded crab were more similar to those with the Lifelike crab than to those with the Translucent crab. Furthermore, octopuses contacted the Black cuboid least among the models. These results suggest that octopuses compositely detect both visual and tactile information in order to perceive an object. Furthermore, octopuses possess the potential priority either for visual or tactile information, by which they process the target object.

Multisensory stimuli shift perceptual priors to facilitate rapid behavior

Multisensory stimuli speed behavioral responses, but the mechanisms subserving these effects remain disputed. Historically, the observation that multisensory reaction times (RTs) outpace models assuming independent sensory channels has been taken as evidence for multisensory integration (the "redundant target effect"; RTE). However, this interpretation has been challenged by alternative explanations based on stimulus sequence effects, RT variability, and/or negative correlations in unisensory processing. To clarify the mechanisms subserving the RTE, we collected RTs from 78 undergraduates in a multisensory simple RT task. Based on previous neurophysiological findings, we hypothesized that the RTE was unlikely to reflect these alternative mechanisms, and more likely reflected pre-potentiation of sensory responses through crossmodal phase-resetting. Contrary to accounts based on stimulus sequence effects, we found that preceding stimuli explained only 3-9% of the variance in apparent RTEs. Comparing three plausible evidence accumulator models, we found that multisensory RT distributions were best explained by increased sensory evidence at stimulus onset. Because crossmodal phase-resetting increases cortical excitability before sensory input arrives, these results are consistent with a mechanism based on pre-potentiation through phase-resetting. Mathematically, this model entails increasing the prior log-odds of stimulus presence, providing a potential link between neurophysiological, behavioral, and computational accounts of multisensory interactions.

Hierarchical drift diffusion modeling uncovers multisensory benefit in numerosity discrimination tasks

Studies of accuracy and reaction time in decision making often observe a speed-accuracy tradeoff, where either accuracy or reaction time is sacrificed for the other. While this effect may mask certain multisensory benefits in performance when accuracy and reaction time are separately measured, drift diffusion models (DDMs) are able to consider both simultaneously. However, drift diffusion models are often limited by large sample size requirements for reliable parameter estimation. One solution to this restriction is the use of hierarchical Bayesian estimation for DDM parameters. Here, we utilize hierarchical drift diffusion models (HDDMs) to reveal a multisensory advantage in auditory-visual numerosity discrimination tasks. By fitting this model with a modestly sized dataset, we also demonstrate that large sample sizes are not necessary for reliable parameter estimation.

Predicting Audiovisual Word Recognition in Noisy Situations: Toward Precision Audiology

OBJECTIVE

Spoken communication is better when one can see as well as hear the talker. Although age-related deficits in speech perception were observed, Tye-Murray and colleagues found that even when age-related deficits in audiovisual (AV) speech perception were observed, AV performance could be accurately predicted from auditory-only (A-only) and visual-only (V-only) performance, and that knowing individuals' ages did not increase the accuracy of prediction. This finding contradicts conventional wisdom, according to which age-related differences in AV speech perception are due to deficits in the integration of auditory and visual information, and our primary goal was to determine whether Tye-Murray et al.'s finding with a closed-set test generalizes to situations more like those in everyday life. A second goal was to test a new predictive model that has important implications for audiological assessment.

DESIGN

Participants (N = 109; ages 22-93 years), previously studied by Tye-Murray et al., were administered our new, open-set Lex-List test to assess their auditory, visual, and audiovisual perception of individual words. All testing was conducted in six-talker babble (three males and three females) presented at approximately 62 dB SPL. The level of the audio for the Lex-List items, when presented, was approximately 59 dB SPL because pilot testing suggested that this signal-to-noise ratio would avoid ceiling performance under the AV condition.

RESULTS

Multiple linear regression analyses revealed that A-only and V-only performance accounted for 87.9% of the variance in AV speech perception, and that the contribution of age failed to reach significance. Our new parabolic model accounted for even more (92.8%) of the variance in AV performance, and again, the contribution of age was not significant. Bayesian analyses revealed that for both linear and parabolic models, the present data were almost 10 times as likely to occur with a reduced model (without age) than with a full model (with age as a predictor). Furthermore, comparison of the two reduced models revealed that the data were more than 100 times as likely to occur with the parabolic model than with the linear regression model.

CONCLUSIONS

The present results strongly support Tye-Murray et al.'s hypothesis that AV performance can be accurately predicted from unimodal performance and that knowing individuals' ages does not increase the accuracy of that prediction. Our results represent an important initial step in extending Tye-Murray et al.'s findings to situations more like those encountered in everyday communication. The accuracy with which speech perception was predicted in this study foreshadows a form of precision audiology in which determining individual strengths and weaknesses in unimodal and multimodal speech perception facilitates identification of targets for rehabilitative efforts aimed at recovering and maintaining speech perception abilities critical to the quality of an older adult's life.

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI - evoked by asynchronous yet temporally flanking irrelevant stimuli - result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2-3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2-3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement - in contrast to the redundant target effect - was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

Visual Perceptual Load Attenuates Age-Related Audiovisual Integration in an Audiovisual Discrimination Task

Previous studies confirmed that the cognitive resources are limited for each person, and perceptual load affects the detection of stimulus greatly; however, how the visual perceptual load influences audiovisual integration (AVI) is still unclear. Here, 20 older and 20 younger adults were recruited to perform an auditory/visual discrimination task under various visual perceptual-load conditions. The analysis for the response times revealed a significantly faster response to the audiovisual stimulus than to the visual stimulus or auditory stimulus (all p < 0.001), and a significantly slower response by the older adults than by the younger adults to all targets (all p ≤ 0.024). The race-model analysis revealed a higher AV facilitation effect for older (12.54%) than for younger (7.08%) adults under low visual perceptual-load conditions; however, no obvious difference was found between younger (2.92%) and older (3.06%) adults under medium visual perceptual-load conditions. Only the AV depression effect was found for both younger and older adults under high visual perceptual-load conditions. Additionally, the peak latencies of AVI were significantly delayed in older adults under all visual perceptual-load conditions. These results suggested that visual perceptual load decreased AVI (i.e., depression effects), and the AVI effect was increased but delayed for older adults.

Seeing Beyond Your Nose? The Effects of Lifelong Olfactory Sensory Deprivation on Cerebral Audio-visual Integration

Lifelong auditory and visual sensory deprivation have been demonstrated to alter both perceptual acuity and the neural processing of remaining senses. Recently, it was demonstrated that individuals with anosmia, i.e. complete olfactory sensory deprivation, displayed enhanced multisensory integration performance. Whether this ability is due to a reorganization of olfactory processing regions to focus on cross-modal multisensory information or whether it is due to enhanced processing within multisensory integration regions is not known. To dissociate these two outcomes, we investigated the neural processing of dynamic audio-visual stimuli in individuals with congenital anosmia and matched controls (both groups, n = 33) using functional magnetic resonance imaging. Specifically, we assessed whether the previously demonstrated multisensory enhancement is related to cross-modal processing of multisensory stimuli in olfactory associated regions, the piriform and olfactory orbitofrontal cortices, or enhanced multisensory processing in established multisensory integration regions, the superior temporal and intraparietal sulci. No significant group differences were found in the a priori hypothesized regions using region of interest analyses. However, exploratory whole-brain analysis suggested higher activation related to multisensory integration within the posterior superior temporal sulcus, in close proximity to the multisensory region of interest, in individuals with congenital anosmia. No group differences were demonstrated in olfactory associated regions. Although results were outside our hypothesized regions, combined, they tentatively suggest that enhanced processing of audio-visual stimuli in individuals with congenital anosmia may be mediated by multisensory, and not primary sensory, cerebral regions.

Schizotypal personality traits and multisensory integration: An investigation using the McGurk effect

Multisensory integration, the process by which sensory information from different sensory modalities are bound together, is hypothesized to contribute to perceptual symptomatology in schizophrenia, in whom multisensory integration differences have been consistently found. Evidence is emerging that these differences extend across the schizophrenia spectrum, including individuals in the general population with higher levels of schizotypal traits. In the current study, we used the McGurk task as a measure of multisensory integration. We measured schizotypal traits using the Schizotypal Personality Questionnaire (SPQ), hypothesizing that higher levels of schizotypal traits, specifically Unusual Perceptual Experiences and Odd Speech subscales, would be associated with decreased multisensory integration of speech. Surprisingly, Unusual Perceptual Experiences were not associated with multisensory integration. However, Odd Speech was associated with multisensory integration, and this association extended more broadly across the Disorganized factor of the SPQ, including Odd or Eccentric Behaviour. Individuals with higher levels of Odd or Eccentric Behaviour scores also demonstrated poorer lip-reading abilities, which partially explained performance in the McGurk task. This suggests that aberrant perceptual processes affecting individuals across the schizophrenia spectrum may relate to disorganized symptomatology.

Multisensory Integration as per Technological Advances: A Review

Multisensory integration research has allowed us to better understand how humans integrate sensory information to produce a unitary experience of the external world. However, this field is often challenged by the limited ability to deliver and control sensory stimuli, especially when going beyond audio-visual events and outside laboratory settings. In this review, we examine the scope and challenges of new technology in the study of multisensory integration in a world that is increasingly characterized as a fusion of physical and digital/virtual events. We discuss multisensory integration research through the lens of novel multisensory technologies and, thus, bring research in human-computer interaction, experimental psychology, and neuroscience closer together. Today, for instance, displays have become volumetric so that visual content is no longer limited to 2D screens, new haptic devices enable tactile stimulation without physical contact, olfactory interfaces provide users with smells precisely synchronized with events in virtual environments, and novel gustatory interfaces enable taste perception through levitating stimuli. These technological advances offer new ways to control and deliver sensory stimulation for multisensory integration research beyond traditional laboratory settings and open up new experimentations in naturally occurring events in everyday life experiences. Our review then summarizes these multisensory technologies and discusses initial insights to introduce a bridge between the disciplines in order to advance the study of multisensory integration.

Neurophysiological Indices of Audiovisual Speech Processing Reveal a Hierarchy of Multisensory Integration Effects

Seeing a speaker's face benefits speech comprehension, especially in challenging listening conditions. This perceptual benefit is thought to stem from the neural integration of visual and auditory speech at multiple stages of processing, whereby movement of a speaker's face provides temporal cues to auditory cortex, and articulatory information from the speaker's mouth can aid recognizing specific linguistic units (e.g., phonemes, syllables). However, it remains unclear how the integration of these cues varies as a function of listening conditions. Here, we sought to provide insight on these questions by examining EEG responses in humans (males and females) to natural audiovisual (AV), audio, and visual speech in quiet and in noise. We represented our speech stimuli in terms of their spectrograms and their phonetic features and then quantified the strength of the encoding of those features in the EEG using canonical correlation analysis (CCA). The encoding of both spectrotemporal and phonetic features was shown to be more robust in AV speech responses than what would have been expected from the summation of the audio and visual speech responses, suggesting that multisensory integration occurs at both spectrotemporal and phonetic stages of speech processing. We also found evidence to suggest that the integration effects may change with listening conditions; however, this was an exploratory analysis and future work will be required to examine this effect using a within-subject design. These findings demonstrate that integration of audio and visual speech occurs at multiple stages along the speech processing hierarchy.SIGNIFICANCE STATEMENT During conversation, visual cues impact our perception of speech. Integration of auditory and visual speech is thought to occur at multiple stages of speech processing and vary flexibly depending on the listening conditions. Here, we examine audiovisual (AV) integration at two stages of speech processing using the speech spectrogram and a phonetic representation, and test how AV integration adapts to degraded listening conditions. We find significant integration at both of these stages regardless of listening conditions. These findings reveal neural indices of multisensory interactions at different stages of processing and provide support for the multistage integration framework.

Integration of vestibular and hindlimb inputs by vestibular nucleus neurons: multisensory influences on postural control

We recently demonstrated in decerebrate and conscious cat preparations that hindlimb somatosensory inputs converge with vestibular afferent input onto neurons in multiple central nervous system (CNS) locations that participate in balance control. Although it is known that head position and limb state modulate postural reflexes, presumably through vestibulospinal and reticulospinal pathways, the combined influence of the two inputs on the activity of neurons in these brainstem regions is unknown. In the present study, we evaluated the responses of vestibular nucleus (VN) neurons to vestibular and hindlimb stimuli delivered separately and together in conscious cats. We hypothesized that VN neuronal firing during activation of vestibular and limb proprioceptive inputs would be well fit by an additive model. Extracellular single-unit recordings were obtained from VN neurons. Sinusoidal whole body rotation in the roll plane was used as the search stimulus. Units responding to the search stimulus were tested for their responses to 10° ramp-and-hold roll body rotation, 60° extension hindlimb movement, and both movements delivered simultaneously. Composite response histograms were fit by a model of low- and high-pass filtered limb and body position signals using least squares nonlinear regression. We found that VN neuronal activity during combined vestibular and hindlimb proprioceptive stimulation in the conscious cat is well fit by a simple additive model for signals with similar temporal dynamics. The mean R² value for goodness of fit across all units was 0.74 ± 0.17. It is likely that VN neurons that exhibit these integrative properties participate in adjusting vestibulospinal outflow in response to limb state.NEW & NOTEWORTHY Vestibular nucleus neurons receive convergent information from hindlimb somatosensory inputs and vestibular inputs. In this study, extracellular single-unit recordings of vestibular nucleus neurons during conditions of passively applied limb movement, passive whole body rotations, and combined stimulation were well fit by an additive model. The integration of hindlimb somatosensory inputs with vestibular inputs at the first stage of vestibular processing suggests that vestibular nucleus neurons account for limb position in determining vestibulospinal responses to postural perturbations.

Audiovisual interactions in speeded discrimination of a visual event

The integration of information from different senses is central to our perception of the external world. Audiovisual interactions have been particularly well studied in this context and various illusions have been developed to demonstrate strong influences of these interactions on the final percept. Using audiovisual paradigms, previous studies have shown that even task-irrelevant information provided by a secondary modality can change the detection and discrimination of a primary target. These modulations have been found to be significantly dependent on the relative timing between auditory and visual stimuli. Although these interactions in time have been commonly reported, we have still limited understanding of the relationship between the modulations of event-related potentials (ERPs) and final behavioral performance. Here, we aimed to shed light on this important issue by using a speeded discrimination paradigm combined with electroencephalogram (EEG). During the experimental sessions, the timing between an auditory click and a visual flash was varied over a wide range of stimulus onset asynchronies and observers were engaged in speeded discrimination of flash location. Behavioral reaction times were significantly changed by click timing. Furthermore, the modulations of evoked activities over medial parietal/parieto-occipital electrodes were associated with this effect. These modulations were within the 126-176 ms time range and more importantly, they were also correlated with the changes in reaction times. These results provide an important functional link between audiovisual interactions at early stages of sensory processing and reaction times. Together with previous research, they further suggest that early crossmodal interactions play a critical role in perceptual performance.

The additive nature of the human multisensory evoked pupil response

Pupillometry has received increased interest for its usefulness in measuring various sensory processes as an alternative to behavioural assessments. This is also apparent for multisensory investigations. Studies of the multisensory pupil response, however, have produced conflicting results. Some studies observed super-additive multisensory pupil responses, indicative of multisensory integration (MSI). Others observed additive multisensory pupil responses even though reaction time (RT) measures were indicative of MSI. Therefore, in the present study, we investigated the nature of the multisensory pupil response by combining methodological approaches of previous studies while using supra-threshold stimuli only. In two experiments we presented auditory and visual stimuli to observers that evoked a(n) (onset) response (be it constriction or dilation) in a simple detection task and a change detection task. In both experiments, the RT data indicated MSI as shown by race model inequality violation. Still, the multisensory pupil response in both experiments could best be explained by linear summation of the unisensory pupil responses. We conclude that the multisensory pupil response for supra-threshold stimuli is additive in nature and cannot be used as a measure of MSI, as only a departure from additivity can unequivocally demonstrate an interaction between the senses.

Semantic Congruency Modulates the Effect of Attentional Load on the Audiovisual Integration of Animate Images and Sounds

Attentional processes play a complex and multifaceted role in the integration of input from different sensory modalities. However, whether increased attentional load disrupts the audiovisual (AV) integration of common objects that involve semantic content remains unclear. Furthermore, knowledge regarding how semantic congruency interacts with attentional load to influence the AV integration of common objects is limited. We investigated these questions by examining AV integration under various attentional-load conditions. AV integration was assessed by adopting an animal identification task using unisensory (animal images and sounds) and AV stimuli (semantically congruent AV objects and semantically incongruent AV objects), while attentional load was manipulated by using a rapid serial visual presentation task. Our results indicate that attentional load did not attenuate the integration of semantically congruent AV objects. However, semantically incongruent animal sounds and images were not integrated (as there was no multisensory facilitation), and the interference effect produced by the semantically incongruent AV objects was reduced by increased attentional-load manipulations. These findings highlight the critical role of semantic congruency in modulating the effect of attentional load on the AV integration of common objects.