The interactions of multisensory integration with endogenous and exogenous attention

Electrophysiological responses of audiovisual integration from infancy to adulthood

Our ability to merge information from different senses into a unified percept is a crucial perceptual process for efficient interaction with our multisensory environment. Yet, the developmental process underlying how the brain implements multisensory integration (MSI) remains poorly known. This cross-sectional study aims to characterize the developmental patterns of audiovisual events in 131 individuals aged from 3 months to 30 years. Electroencephalography (EEG) was recorded during a passive task, including simple auditory, visual, and audiovisual stimuli. In addition to examining age-related variations in MSI responses, we investigated Event-Related Potentials (ERPs) linked with auditory and visual stimulation alone. This was done to depict the typical developmental trajectory of unisensory processing from infancy to adulthood within our sample and to contextualize the maturation effects of MSI in relation to unisensory development. Comparing the neural response to audiovisual stimuli to the sum of the unisensory responses revealed signs of MSI in the ERPs, more specifically between the P2 and N2 components (P2 effect). Furthermore, adult-like MSI responses emerge relatively late in the development, around 8 years old. The automatic integration of simple audiovisual stimuli is a long developmental process that emerges during childhood and continues to mature during adolescence with ERP latencies decreasing with age.

Influence of flavor information on visual search: Attentional capture by and suppression of flavor-associated colors

Numerous studies have demonstrated the impact of flavor cues on visual search, yet the underlying mechanisms remain elusive. In this experiment, we used event-related potentials (ERPs) to examine whether, and if so, how flavor information could lead to attentional capture by, and suppression of, flavor-associated colors. The participants were asked to taste certain flavored beverages and subsequently complete a shape-based visual search task, while their neural activities were simultaneously recorded. The behavioral results revealed that the participants made slower responses when a distractor in the flavor-associated color (DFAC) was present, suggesting an attentional bias toward the flavor-associated color. The ERP results revealed that the N2pc was detected if the target and the DFAC were shown in the same visual field (e.g. both target and DFCA on the right side of the screen), when the pairings between flavor cues and target colors were incongruent. However, the N2pc was not observed if the target and the DFAC were shown in the opposite visual fields (e.g. target on the right and DFCA on the left side of the screen) for the incongruent color-flavor pairings. Moreover, the distractor positivity (Pd) was observed if the target and the DFAC were shown in the opposite visual field for the congruent color-flavor pairings. These results suggest that both attentional capture and suppression are involved in the influence of flavor information on visual search. Collectively, these findings provide initial electrophysiological evidence on the mechanisms of the crossmodal influence of flavor cues on visual search.

The stimulus-driven and representation-driven cross-modal attentional spreading are both modulated by audiovisual temporal synchrony

Multisensory integration and attention can interact in a way that attention to the visual constituent of a multisensory object results in an attentional spreading to its ignored auditory constituent, which can be either stimulus-driven or representation-driven depending on whether the object's visual constituent receives extra representation-based selective attention. Previous research using simple unrelated audiovisual combinations has shown that the stimulus-driven attentional spreading is contingent on audiovisual temporal simultaneity. However, little is known about whether this temporal constraint applies also to the representation-driven attentional spreading, and whether it holds for the stimulus-driven process elicited by real-life multisensory objects. The current event-related potential study investigated these questions by systematically manipulating the visual-to-auditory stimulus onset asynchrony (SOA: 0/100/300 ms) in an object-selective visual recognition task wherein the representation-driven and stimulus-driven spreading processes, measured as two distinct auditory negative difference (Nd) components, could be isolated independently. Our results showed that both the representation-driven and stimulus-driven Nds decreased as the SOA increased. Interestingly, the representation-driven Nd was completely absent, whereas the stimulus-driven Nd was still robust, when the auditory constituents were delayed by 300 ms. These findings not only indicate that the role of audiovisual simultaneity in the representation-driven attentional spreading has been underestimated, but also suggest that learned associations between the unisensory constituents of real-life objects render the stimulus-driven attentional spreading more tolerant of audiovisual asynchrony.

Pitch Improvement in Attentional Blink: A Study across Audiovisual Asymmetries

Attentional blink (AB) is a phenomenon in which the perception of a second target is impaired when it appears within 200-500 ms after the first target. Sound affects an AB and is accompanied by the appearance of an asymmetry during audiovisual integration, but it is not known whether this is related to the tonal representation of sound. The aim of the present study was to investigate the effect of audiovisual asymmetry on attentional blink and whether the presentation of pitch improves the ability to detect a target during an AB that is accompanied by audiovisual asymmetry. The results showed that as the lag increased, the subject's target recognition improved and the pitch produced further improvements. These improvements exhibited a significant asymmetry across the audiovisual channel. Our findings could contribute to better utilizations of audiovisual integration resources to improve attentional transients and auditory recognition decline, which could be useful in areas such as driving and education.

Spatial attention modulates multisensory integration: The dissociation between exogenous and endogenous orienting

Previous studies have separately found that exogenous orienting decreases multisensory integration (MSI), while endogenous orienting enhances MSI. It is currently unclear, however, why the two orientations have opposite effects on MSI. In the current study, we investigated the interaction between spatial attention and MSI in two experiments based on the cue-target paradigm. Experiment 1 separated exogenous and endogenous orienting to investigate the effect of spatial attention on MSI by varying the predictability of the cue. Experiment 2 further explored the effect of endogenous orienting on MSI. We found that exogenous orienting induced by the directionality of the cue decreased MSI, while endogenous orienting induced by the predictability of the cue enhanced MSI. The role of spatial orienting need and spatial attention bias in the modulation of MSI by exogenous and endogenous orienting was discussed. The present study sheds new light on how spatial attention modulates MSI processes.

Enhancing learning outcomes through multisensory integration: A fMRI study of audio-visual training in virtual reality

The integration of information from different sensory modalities is a fundamental process that enhances perception and performance in real and virtual environments (VR). Understanding these mechanisms, especially during learning tasks that exploit novel multisensory cue combinations provides opportunities for the development of new rehabilitative interventions. This study aimed to investigate how functional brain changes support behavioural performance improvements during an audio-visual (AV) learning task. Twenty healthy participants underwent a 30 min daily VR training for four weeks. The task was an AV adaptation of a 'scanning training' paradigm that is commonly used in hemianopia rehabilitation. Functional magnetic resonance imaging (fMRI) and performance data were collected at baseline, after two and four weeks of training, and four weeks post-training. We show that behavioural performance, operationalised as mean reaction time reduction in VR, significantly improves. In separate tests in a controlled laboratory environment, we showed that the behavioural performance gains in the VR training environment transferred to a significant mean RT reduction for the trained AV voluntary task on a computer screen. Enhancements were observed in both the visual-only and AV conditions, with the latter demonstrating a faster response time supported by the presence of audio cues. The behavioural learning effect also transfers to two additional tasks that were tested: a visual search task and an involuntary visual task. Our fMRI results reveal an increase in functional activation (BOLD signal) in multisensory brain regions involved in early-stage AV processing: the thalamus, the caudal inferior parietal lobe and cerebellum. These functional changes were only observed for the trained, multisensory, task and not for unimodal visual stimulation. Functional activation changes in the thalamus were significantly correlated to behavioural performance improvements. This study demonstrates that incorporating spatial auditory cues to voluntary visual training in VR leads to augmented brain activation changes in multisensory integration, resulting in measurable performance gains across tasks. The findings highlight the potential of VR-based multisensory training as an effective method for enhancing cognitive function and as a potentially valuable tool in rehabilitative programmes.

Different levels of prepulse inhibition among patients with first-episode schizophrenia, bipolar disorder and major depressive disorder

BACKGROUND

Deficits in prepulse inhibition may be a common feature in first-episode schizophrenia, bipolar disorder (BD) and major depressive disorder (MDD). We sought to explore the levels and viability of prepulse inhibition to differentiate first-episode schizophrenia, BD and MDD in patient populations.

METHODS

We tested patients with first-episode schizophrenia, BD or MDD and healthy controls using prepulse inhibition paradigms, namely perceived spatial co-location (PSC-PPI) and perceived spatial separation (PSS-PPI).

RESULTS

We included 53 patients with first-episode schizophrenia, 30 with BD and 25 with MDD, as well as 82 healthy controls. The PSS-PPI indicated that the levels of prepulse inhibition were smallest to largest, respectively, in the first-episode schizophrenia, BD, MDD and control groups. Relative to the healthy controls, the prepulse inhibition deficits in the first-episode schizophrenia group were significant (p < 0.001), but the prepulse inhibitions were similar between patients with BD and healthy controls, and between patients with MDD and healthy controls. The receiver operating characteristic curve analysis showed that PSS-PPI (area under the curve [AUC] 0.73, p < 0.001) and latency (AUC 0.72, p < 0.001) were significant for differentiating patients with first-episode schizophrenia or BD from healthy controls.

LIMITATIONS

The demographics of the 4 groups were not ideally matched. We did not perform cognitive assessments. The possible confounding effect of medications on prepulse inhibition could not be eliminated.

CONCLUSION

The level of prepulse inhibition among patients with first-episode schizophrenia was the lowest, with levels among patients with BD, patients with MDD and healthy controls increasingly higher. The PSS-PPI paradigm was more effective than PSC-PPI to recognize deficits in prepulse inhibition. These results provide a basis for further research on biological indicators that can assist differential diagnoses in psychosis.

The neural oscillations in delta- and theta-bands contribute to divided attention in audiovisual integration

One of key mechanisms implicated in multisensory processing is neural oscillations in distinct frequency band. Many studies explored the modulation of attention by recording the electroencephalography signals when subjects attended one modality, and ignored the other modality input. However, when attention is directed toward one modality, it may be not always possible to shut out completely inputs from a different modality. Since many situations require division of attention between audition and vision, it is imperative to investigate the neural mechanisms underlying processing of concurrent auditory and visual sensory streams. In the present study, we designed a task of audiovisual semantic discrimination, in which the subjects were asked to share attention to both auditory and visual stimuli. We explored the contribution of neural oscillations in lower-frequency to the modulation of divided attention on audiovisual integration. Our results implied that theta-band activity contributes to the early modulation of divided attention, and delta-band activity contributes to the late modulation of divided attention to audiovisual integration. Moreover, the fronto-central delta- and theta-bands activity is likely a marker of divided attention in audiovisual integration, and the neural oscillation on delta- and theta-bands is conducive to allocating attention resources to dual-tasking involving task-coordinating abilities.

Non-spatial inhibition of return attenuates audiovisual integration owing to modality disparities

Although previous studies have investigated the relationship between inhibition of return (IOR) and multisensory integration, the influence of non-spatial has not been explored. The present study aimed to investigate the influence of non-spatial IOR on audiovisual integration by using a "prime-neutral cue-target" paradigm. In Experiment 1, which manipulated prime validity and target modality, the targets were positioned centrally, revealing significant non-spatial IOR effects in the visual, auditory, and audiovisual modalities. Analysis of relative multisensory response enhancement (rMRE) indicated substantial audiovisual integration enhancement in both valid and invalid target conditions. Furthermore, the enhancement was weaker for valid targets than for invalid targets. In Experiment 2, the targets were positioned above and below to rule out repetition blindness (RB); this experiment successfully replicated the results observed in Experiment 1. Notably, Experiments 1 and 2 consistently found that the correlation between modality differences and rMRE for valid targets indicated that differences in signal strength between visual and auditory modalities contributed to a reduction in audiovisual integration. However, the absence of correlation with the invalid target suggests that attention, as a key factor, may play a significant role in this process. The present study highlights how non-spatial IOR reduces audiovisual integration and sheds light on the complex interaction between attention and multisensory integration.

Multisensory integration augmenting motor processes among older adults

Objective

Multisensory integration enhances sensory processing in older adults. This study aimed to investigate how the sensory enhancement would modulate the motor related process in healthy older adults.

Method

Thirty-one older adults (12 males, mean age 67.7 years) and 29 younger adults as controls (16 males, mean age 24.9 years) participated in this study. Participants were asked to discriminate spatial information embedded in the unisensory (visual or audial) and multisensory (audiovisual) conditions. The responses made by the movements of the left and right wrists corresponding to the spatial information were registered with specially designed pads. The electroencephalogram (EEG) marker was the event-related super-additive P2 in the frontal-central region, the stimulus-locked lateralized readiness potentials (s-LRP) and response-locked lateralized readiness potentials (r-LRP).

Results

Older participants showed significantly faster and more accurate responses than controls in the multisensory condition than in the unisensory conditions. Both groups had significantly less negative-going s-LRP amplitudes elicited at the central sites in the between-condition contrasts. However, only the older group showed significantly less negative-going, centrally distributed r-LRP amplitudes. More importantly, only the r-LRP amplitude in the audiovisual condition significantly predicted behavioral performance.

Conclusion

Audiovisual integration enhances reaction time, which associates with modulated motor related processes among the older participants. The super-additive effects modulate both the motor preparation and generation processes. Interestingly, only the modulated motor generation process contributes to faster reaction time. As such effects were observed in older but not younger participants, multisensory integration likely augments motor functions in those with age-related neurodegeneration.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

The modulation of expectation violation on attention: Evidence from the spatial cueing effects

The study sought to investigate whether and how expectation violation can modulate attention using the exogenous spatial cueing paradigm, under the theoretical framework of the Memory Encoding Cost (MEC) model. The MEC proposes that exogenous spatial cueing effects are mainly driven by a combination of two distinct mechanisms: attentional facilitation triggered by the presence of an abrupt cue, and attentional suppression induced by memory encoding of the cue. In current experiments, participants needed to identify a target letter that was sometimes preceded by a peripheral onset cue. Various types of expectation violation were introduced by regulating the probability of cue presentation (Experiments 1 & 5), the probability of cue location (Experiments 2 & 4), and the probability of irrelevant sound presentation (Experiment 3). The results showed that expectation violation could enhance the cueing effect (valid vs. invalid cue) in some cases. More crucially, all experiments consistently observed asymmetrical modulation of expectation violation on the cost (invalid vs. neutral cue) and benefit (valid vs. neutral cue) effects: Expectation violation increased the cost effects while did not modulate or decreased (or even reversed) the benefit effects. Furthermore, Experiment 5 provided direct evidence that violation of expectations could enhance the memory encoding of a cue (e.g., color) and this memory advantage could manifest quickly in the early stages of the experiment. The MEC better explains these findings than some traditional models like the spotlight: Expectation violation can both enhance the attentional facilitation of the cue and memory encoding of irrelevant cue information. These findings suggest that expectation violation has a general adaptive function in modulating the attention selectivity.

The effects of attention in auditory-visual integration revealed by time-varying networks

Attention and audiovisual integration are crucial subjects in the field of brain information processing. A large number of previous studies have sought to determine the relationship between them through specific experiments, but failed to reach a unified conclusion. The reported studies explored the relationship through the frameworks of early, late, and parallel integration, though network analysis has been employed sparingly. In this study, we employed time-varying network analysis, which offers a comprehensive and dynamic insight into cognitive processing, to explore the relationship between attention and auditory-visual integration. The combination of high spatial resolution functional magnetic resonance imaging (fMRI) and high temporal resolution electroencephalography (EEG) was used. Firstly, a generalized linear model (GLM) was employed to find the task-related fMRI activations, which was selected as regions of interesting (ROIs) for nodes of time-varying network. Then the electrical activity of the auditory-visual cortex was estimated via the normalized minimum norm estimation (MNE) source localization method. Finally, the time-varying network was constructed using the adaptive directed transfer function (ADTF) technology. Notably, Task-related fMRI activations were mainly observed in the bilateral temporoparietal junction (TPJ), superior temporal gyrus (STG), primary visual and auditory areas. And the time-varying network analysis revealed that V1/A1↔STG occurred before TPJ↔STG. Therefore, the results supported the theory that auditory-visual integration occurred before attention, aligning with the early integration framework.

Visual adaptation changes the susceptibility to the fission illusion

Sound-induced flash illusion (SiFI) is the illusion that participants perceive incorrectly that the number of visual flashes is equal to the number of auditory beeps when presented within 100 ms. Although previous studies found that repetition suppression can reduce an individual's perceptual sensitivity to the SiFI, there is not yet a consensus as to how visual adaptation affects the SiFI. In the present study, we added prolonged adapting visual stimuli prior to the presentation of audiovisual stimuli to investigate whether the bottom-up factor of adaptation affects the SiFI. The adapting visual stimuli consisted of one or two of the same visual stimuli that lasted for 2 minutes in succession, followed by the audiovisual stimuli. Both adaptation conditions showed SiFI effects. The accuracy of adapting double-flashes was significantly lower than that of in adapting a single flash for the fission illusion. Our analyses indicated that such a pattern could be attributed to a lower d' in adapting double-flashes than in adapting a single flash. However, the accuracy, discriminability and criterion were not significantly different between the two adaptation conditions because of the instability of the fusion illusion. Thus, the present study indicated that the reduced perceptual sensitivity based on visual adaptation could enhance the fission illusion in multisensory integration.

The integration of continuous audio and visual speech in a cocktail-party environment depends on attention

In noisy environments, our ability to understand speech benefits greatly from seeing the speaker's face. This is attributed to the brain's ability to integrate audio and visual information, a process known as multisensory integration. In addition, selective attention plays an enormous role in what we understand, the so-called cocktail-party phenomenon. But how attention and multisensory integration interact remains incompletely understood, particularly in the case of natural, continuous speech. Here, we addressed this issue by analyzing EEG data recorded from participants who undertook a multisensory cocktail-party task using natural speech. To assess multisensory integration, we modeled the EEG responses to the speech in two ways. The first assumed that audiovisual speech processing is simply a linear combination of audio speech processing and visual speech processing (i.e., an A+V model), while the second allows for the possibility of audiovisual interactions (i.e., an AV model). Applying these models to the data revealed that EEG responses to attended audiovisual speech were better explained by an AV model, providing evidence for multisensory integration. In contrast, unattended audiovisual speech responses were best captured using an A+V model, suggesting that multisensory integration is suppressed for unattended speech. Follow up analyses revealed some limited evidence for early multisensory integration of unattended AV speech, with no integration occurring at later levels of processing. We take these findings as evidence that the integration of natural audio and visual speech occurs at multiple levels of processing in the brain, each of which can be differentially affected by attention.

Aging effect of cross-modal interactions during audiovisual detection and discrimination by behavior and ERPs

Introduction

Numerous studies have shown that aging greatly affects audiovisual integration; however, it is still unclear when the aging effect occurs, and its neural mechanism has yet to be fully elucidated.

Methods

We assessed the audiovisual integration (AVI) of older (n = 40) and younger (n = 45) adults using simple meaningless stimulus detection and discrimination tasks. The results showed that the response was significantly faster and more accurate for younger adults than for older adults in both the detection and discrimination tasks. The AVI was comparable for older and younger adults during stimulus detection (9.37% vs. 9.43%); however, the AVI was lower for older than for younger adults during stimulus discrimination (9.48% vs. 13.08%) behaviorally. The electroencephalography (EEG) analysis showed that comparable AVI amplitude was found at 220-240 ms for both groups during stimulus detection and discrimination, but there was no significant difference between brain regions for older adults but a higher AVI amplitude in the right posterior for younger adults. Additionally, a significant AVI was found for younger adults in 290-310 ms but was absent for older adults during stimulus discrimination. Furthermore, significant AVI was found in the left anterior and right anterior at 290-310 ms for older adults but in the central, right posterior and left posterior for younger adults.

Discussion

These results suggested that the aging effect of AVI occurred in multiple stages, but the attenuated AVI mainly occurred in the later discriminating stage attributed to attention deficit.

The role of primary motor cortex in manual inhibition of return: A transcranial magnetic stimulation study

Inhibition of return (IOR) is a behavioural phenomenon characterised by longer response times (RTs) to stimuli presented at previously cued versus uncued locations. The neural mechanisms underlying IOR effects are not fully understood. Previous neurophysiological studies have identified a role of frontoparietal areas including posterior parietal cortex (PPC) in the generation of IOR, but the contribution of primary motor cortex (M1) has not been directly tested. The present study investigated the effects of single-pulse transcranial magnetic stimulation (TMS) over M1 on manual IOR in a key-press task where peripheral (left or right) targets followed a cue at the same or opposite location at different SOAs (100/300/600/1000 ms). In Experiment 1, TMS was applied over right M1 on a randomized 50% of trials. In Experiment 2, active or sham stimulation was provided in separate blocks. In the absence of TMS (non-TMS trials in Experiment 1 and sham trials in Experiment 2), evidence of IOR was observed in RTs at longer SOAs. In both experiments, IOR effects differed between TMS and non-TMS/sham conditions, but the effects of TMS were greater and statistically significant in Experiment 1 where TMS and non-TMS trials were randomly interspersed. The magnitude of motor-evoked potentials was not altered by the cue-target relationship in either experiment. These findings do not support a key role of M1 in the mechanisms of IOR but suggest the need for further research to elucidate the role of the motor system in manual IOR effects.

Sustained visual attentional load modulates audiovisual integration in older and younger adults

Previous studies have shown that attention influences audiovisual integration (AVI) in multiple stages, but it remains unclear how AVI interacts with attentional load. In addition, while aging has been associated with sensory-functional decline, little is known about how older individuals integrate cross-modal information under attentional load. To investigate these issues twenty older adults and 20 younger adults were recruited to conduct a dual task including a multiple object tracking (MOT) task, which manipulated sustained visual attentional load, and an audiovisual discrimination task, which assesses AVI. The results showed that response times were shorter and hit rate was higher for audiovisual stimuli than for auditory or visual stimuli alone and in younger adults than in older adults. The race model analysis showed that AVI was higher under the load_3 condition (monitoring two targets of the MOT task) than under any other load condition (no-load [NL], one or three targets monitoring). This effect was found regardless of age. However, AVI was lower in older adults than younger adults under NL condition. Moreover, the peak latency was longer, and the time window of AVI was delayed in older adults compared to younger adults under all conditions. These results suggest that slight visual sustained attentional load increased AVI but that heavy visual sustained attentional load decreased AVI, which supports the claim that attention resource was limited, and we further proposed that AVI was positively modulated by attentional resource. Finally, there were substantial impacts of aging on AVI; AVI was delayed in older adults.

Effect of anticipatory multisensory integration on sensory-motor performance

Multisensory integration (MSI) is a phenomenon that occurs in sensory areas after the presentation of multimodal stimuli. Nowadays, little is known about the anticipatory top-down processes taking place in the preparation stage of processing before the stimulus onset. Considering that the top-down modulation of modality-specific inputs might affect the MSI process, this study attempts to understand whether the direct modulation of the MSI process, beyond the well-known sensory effects, may lead to additional changes in multisensory processing also in non-sensory areas (i.e., those related to task preparation and anticipation). To this aim, event-related potentials (ERPs) were analyzed both before and after auditory and visual unisensory and multisensory stimuli during a discriminative response task (Go/No-go type). Results showed that MSI did not affect motor preparation in premotor areas, while cognitive preparation in the prefrontal cortex was increased and correlated with response accuracy. Early post-stimulus ERP activities were also affected by MSI and correlated with response time. Collectively, the present results point to the plasticity accommodating nature of the MSI processes, which are not limited to perception and extend to anticipatory cognitive preparation for task execution. Further, the enhanced cognitive control emerging during MSI is discussed in the context of Bayesian accounts of augmented predictive processing related to increased perceptual uncertainty.

Modal-based attention modulates the redundant-signals effect: Role of unimodal target probability

Multisensory integration includes two behavioral manifestations: the modality dominance effect and the redundant-signals effect (RSE). RSE is a multisensory improvement effect in which individuals respond more quickly and accurately to bimodal audiovisual (AV) targets than to unimodal auditory (A) or visual (V) targets. Previous studies have confirmed that RSE is the product of modality interactions between different modalities. The goal of this study was to systematically investigate the effects of the modality dominance manipulated by modal-based attention and unimodal target probability on RSE. The results showed that when paying attention to both the A and V modalities (Exp. 1), RSE was not significantly different between unimodal target probabilities. When selectively paying attention to the A modality (Exp. 2A), RSE was also not significantly different between unimodal target probabilities. However, when selectively paying attention to the V modality (Exp. 2B), the magnitude of RSE showed a significant decreasing trend with the increasing probability of V targets. Our study is the first to reveal that the unimodal target probability significantly modulates RSE in visual selective attention, and this modulatory effect of the unimodal target probability on RSE is opposite to the modulatory effect on the modality dominance effect.

Increases in sensory noise predict attentional disruptions to audiovisual speech perception

We receive information about the world around us from multiple senses which combine in a process known as multisensory integration. Multisensory integration has been shown to be dependent on attention; however, the neural mechanisms underlying this effect are poorly understood. The current study investigates whether changes in sensory noise explain the effect of attention on multisensory integration and whether attentional modulations to multisensory integration occur via modality-specific mechanisms. A task based on the McGurk Illusion was used to measure multisensory integration while attention was manipulated via a concurrent auditory or visual task. Sensory noise was measured within modality based on variability in unisensory performance and was used to predict attentional changes to McGurk perception. Consistent with previous studies, reports of the McGurk illusion decreased when accompanied with a secondary task; however, this effect was stronger for the secondary visual (as opposed to auditory) task. While auditory noise was not influenced by either secondary task, visual noise increased with the addition of the secondary visual task specifically. Interestingly, visual noise accounted for significant variability in attentional disruptions to the McGurk illusion. Overall, these results strongly suggest that sensory noise may underlie attentional alterations to multisensory integration in a modality-specific manner. Future studies are needed to determine whether this finding generalizes to other types of multisensory integration and attentional manipulations. This line of research may inform future studies of attentional alterations to sensory processing in neurological disorders, such as Schizophrenia, Autism, and ADHD.

Differences in eccentricity for sound-induced flash illusion in four visual fields

A sound-induced flash illusion (SiFI) is a multisensory illusion dominated by auditory stimuli, in which the individual perceives that the number of visual flashes is equal to the number of auditory stimuli when visual flashes are presented along with an unequal number of auditory stimuli. Although the mechanisms underlying fission and fusion illusions have been documented, there is not yet a consensus on how they vary according to the different eccentricities. In the present study, by incorporating the classic SiFI paradigm into four different eccentricities, we aimed to investigate whether the SiFI varies under the different eccentricities. The results showed that the fission illusion varied significantly across the four eccentricities, with the perifovea (7°) and peripheral (11°) illusions being greater than the fovea and parafovea (3°) illusions. In contrast, the fusion illusion did not vary significantly across the four eccentricities. Our findings revealed that SiFI was affected by different visual fields and that there were differences between the fission and the fusion illusions. Furthermore, by examining the SiFI of eccentricity across visual fields, this study also suggests that bottom-up factors affect the SiFI.

Effects of audiovisual interactions on working memory: Use of the combined N-back + Go/NoGo paradigm

Background

Approximately 94% of sensory information acquired by humans originates from the visual and auditory channels. Such information can be temporarily stored and processed in working memory, but this system has limited capacity. Working memory plays an important role in higher cognitive functions and is controlled by central executive function. Therefore, elucidating the influence of the central executive function on information processing in working memory, such as in audiovisual integration, is of great scientific and practical importance.

Purpose

This study used a paradigm that combined N-back and Go/NoGo tasks, using simple Arabic numerals as stimuli, to investigate the effects of cognitive load (modulated by varying the magnitude of N) and audiovisual integration on the central executive function of working memory as well as their interaction.

Methods

Sixty college students aged 17-21 years were enrolled and performed both unimodal and bimodal tasks to evaluate the central executive function of working memory. The order of the three cognitive tasks was pseudorandomized, and a Latin square design was used to account for order effects. Finally, working memory performance, i.e., reaction time and accuracy, was compared between unimodal and bimodal tasks with repeated-measures analysis of variance (ANOVA).

Results

As cognitive load increased, the presence of auditory stimuli interfered with visual working memory by a moderate to large extent; similarly, as cognitive load increased, the presence of visual stimuli interfered with auditory working memory by a moderate to large effect size.

Conclusion

Our study supports the theory of competing resources, i.e., that visual and auditory information interfere with each other and that the magnitude of this interference is primarily related to cognitive load.

Effects of exogenous and endogenous cues on attentional orienting in deaf adults

Adults who are deaf have been shown to have better visual attentional orienting than those with typical hearing, especially when the target is located in the periphery of the visual field. However, most studies in this population have assessed exogenous visual attention orienting (bottom-up processing of external cues) rather than endogenous visual attention orienting (top-down processing of internal cues). We used a target detection task to assess both types of visual attention orienting. A modified cue-target paradigm was adopted to assess the facilitation effects of exogenous and endogenous cues during short and long inter-stimulus intervals (ISI), using a 2 (Group: deaf/typically hearing) * 2 (Location: central/peripheral) * 2 (Cue Type: exogenous/endogenous) mixed factorial design. ANOVAs showed that both exogenous cues and endogenous cues can facilitate deaf adults’ visual attentional orienting, and the facilitation effect of exogenous cues on attention orienting was significantly stronger for deaf participants than hearing participants. When the ISI was long, the effect was significantly stronger when the exogenous cue appeared in the periphery of the visual field. In the periphery, deaf adults benefited most from exogenous cues, whereas hearing adults benefited most from endogenous cues. The results suggest that not only exogenous cues but also endogenous cues can facilitate deaf adults’ visual attentional orienting. However, the effect of exogenous cues appears to be greater, especially when the stimulus appears in the peripheral visual field.

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention

It has been reported that attending to stimuli in visual modality can spread to task-irrelevant but synchronously presented stimuli in auditory modality, a phenomenon termed the cross-modal spread of attention, which could be either stimulus-driven or representation-driven depending on whether the visual constituent of an audiovisual object is further selected based on the object representation. The stimulus-driven spread of attention occurs whenever a task-irrelevant sound synchronizes with an attended visual stimulus, regardless of the cross-modal semantic congruency. The present study recorded event-related potentials (ERPs) to investigate whether the stimulus-driven cross-modal spread of attention could be modulated by audio-visual emotional congruency in a visual oddball task where emotion (positive/negative) was task-irrelevant. The results first demonstrated a prominent stimulus-driven spread of attention regardless of audio-visual emotional congruency by showing that for all audiovisual pairs, the extracted ERPs to the auditory constituents of audiovisual stimuli within the time window of 200–300 ms were significantly larger than ERPs to the same auditory stimuli delivered alone. However, the amplitude of this stimulus-driven auditory Nd component during 200–300 ms was significantly larger for emotionally incongruent than congruent audiovisual stimuli when their visual constituents’ emotional valences were negative. Moreover, the Nd was sustained during 300–400 ms only for the incongruent audiovisual stimuli with emotionally negative visual constituents. These findings suggest that although the occurrence of the stimulus-driven cross-modal spread of attention is independent of audio-visual emotional congruency, its magnitude is nevertheless modulated even when emotion is task-irrelevant.

Evaluating attention deficit hyperactivity disorder symptoms in children and adolescents through tracked head movements in a virtual reality classroom: The effect of social cues with different sensory modalities

Background

Attention deficit hyperactivity disorder (ADHD) is clinically diagnosed; however, quantitative analysis to statistically analyze the symptom severity of children with ADHD via the measurement of head movement is still in progress. Studies focusing on the cues that may influence the attention of children with ADHD in classroom settings, where children spend a considerable amount of time, are relatively scarce. Virtual reality allows real-life simulation of classroom environments and thus provides an opportunity to test a range of theories in a naturalistic and controlled manner. The objective of this study was to investigate the correlation between participants’ head movements and their reports of inattention and hyperactivity, and to investigate how their head movements are affected by different social cues of different sensory modalities.

Methods

Thirty-seven children and adolescents with (n = 20) and without (n = 17) ADHD were recruited for this study. All participants were assessed for diagnoses, clinical symptoms, and self-reported symptoms. A virtual reality-continuous performance test (VR-CPT) was conducted under four conditions: (1) control, (2) no-cue, (3) visual cue, and (4) visual/audio cue. A quantitativecomparison of the participants’ head movements was conducted in three dimensions (pitch [head nods], yaw [head turns], and roll [lateral head inclinations]) using a head-mounted display (HMD) in a VR classroom environment. Task-irrelevant head movements were analyzed separately, considering the dimension of movement needed to perform the VR-CPT.

Results

The magnitude of head movement, especially task-irrelevant head movement, significantly correlated with the current standard of clinical assessment in the ADHD group. Regarding the four conditions, head movement showed changes according to the complexity of social cues in both the ADHD and healthy control (HC) groups.

Conclusion

Children and adolescents with ADHD showed decreasing task-irrelevant movements in the presence of social stimuli toward the intended orientation. As a proof-of-concept study, this study preliminarily identifies the potential of VR as a tool to understand and investigate the classroom behavior of children with ADHD in a controlled, systematic manner.

Are auditory cues special? Evidence from cross-modal distractor-induced blindness

A target that shares features with preceding distractor stimuli is less likely to be detected due to a distractor-driven activation of a negative attentional set. This transient impairment in perceiving the target (distractor-induced blindness/deafness) can be found within vision and audition. Recently, the phenomenon was observed in a cross-modal setting involving an auditory target and additional task-relevant visual information (cross-modal distractor-induced deafness). In the current study, consisting of three behavioral experiments, a visual target, indicated by an auditory cue, had to be detected despite the presence of visual distractors. Multiple distractors consistently led to reduced target detection if cue and target appeared in close temporal proximity, confirming cross-modal distractor-induced blindness. However, the effect on target detection was reduced compared to the effect of cross-modal distractor-induced deafness previously observed for reversed modalities. The physical features defining cue and target could not account for the diminished distractor effect in the current cross-modal task. Instead, this finding may be attributed to the auditory cue acting as an especially efficient release signal of the distractor-induced inhibition. Additionally, a multisensory enhancement of visual target detection by the concurrent auditory signal might have contributed to the reduced distractor effect.

Look and ye shall hear: Selective auditory attention modulates the audiovisual correspondence effect

One of the unresolved questions in multisensory research is that of automaticity of consistent associations between sensory features from different modalities (e.g. high visual locations associated with high sound pitch). We addressed this issue by examining a possible role of selective attention in the audiovisual correspondence effect. We orthogonally manipulated loudness and pitch, directing participants’ attention to the auditory modality only and using pitch and loudness identification tasks. Visual stimuli in high, low or central spatial locations appeared simultaneously with the sounds. If the correspondence effect is automatic, it should not be affected by task changes. The results, however, demonstrated a cross-modal pitch-verticality correspondence effect only when participants’ attention was directed to pitch, but not to loudness identification task; moreover, the effect was present only in the upper location. The findings underscore the involvement of selective attention in cross-modal associations and support a top-down account of audiovisual correspondence effects.

The Role of the Interaction between the Inferior Parietal Lobule and Superior Temporal Gyrus in the Multisensory Go/No-go Task

Information from multiple sensory modalities interacts. Using functional magnetic resonance imaging (fMRI), we aimed to identify the neural structures correlated with how cooccurring sound modulates the visual motor response execution. The reaction time (RT) to audiovisual stimuli was significantly faster than the RT to visual stimuli. Signal detection analyses showed no significant difference in the perceptual sensitivity (d') between audiovisual and visual stimuli, while the response criteria (β or c) of the audiovisual stimuli was decreased compared to the visual stimuli. The functional connectivity between the left inferior parietal lobule (IPL) and bilateral superior temporal gyrus (STG) was enhanced in Go processing compared with No-go processing of audiovisual stimuli. Furthermore, the left precentral gyrus (PreCG) showed enhanced functional connectivity with the bilateral STG and other areas of the ventral stream in Go processing compared with No-go processing of audiovisual stimuli. These results revealed that the neuronal network correlated with modulations of the motor response execution after the presentation of both visual stimuli along with cooccurring sound in a multisensory Go/Nogo task, including the left IPL, left PreCG, bilateral STG and some areas of the ventral stream. The role of the interaction between the IPL and STG in transforming audiovisual information into motor behavior is discussed. The current study provides a new perspective for exploring potential brain mechanisms underlying how humans execute appropriate behaviors on the basis of multisensory information.

Multisensory integration attenuates visually induced oculomotor inhibition of return

Inhibition of return (IOR) is a mechanism of the attention system involving bias toward novel stimuli and delayed generation of responses to targets at previously attended locations. According to the two-component theory, IOR consists of a perceptual component and an oculomotor component (oculomotor IOR [O-IOR]) depending on whether the eye movement system is activated. Previous studies have shown that multisensory integration weakens IOR when paying attention to both visual and auditory modalities. However, it remains unclear whether the O-IOR effect attenuated by multisensory integration also occurs when the oculomotor system is activated. Here, using two eye movement experiments, we investigated the effect of multisensory integration on O-IOR using the exogenous spatial cueing paradigm. In Experiment 1, we found a greater visual O-IOR effect compared with audiovisual and auditory O-IOR in divided modality attention. The relative multisensory response enhancement (rMRE) and violations of Miller's bound showed a greater magnitude of multisensory integration in the cued location compared with the uncued location. In Experiment 2, the magnitude of the audiovisual O-IOR effect was significantly less than that of the visual O-IOR in single visual modality selective attention. Implications for the effect of multisensory integration on O-IOR were discussed under conditions of oculomotor system activation, shedding new light on the two-component theory of IOR.

The effect of movement speed on audiovisual temporal integration in streaming-bouncing illusion

Motion perception in real situations is often stimulated by multisensory information. Speed is an essential characteristic of moving objects; however, at present, it is not clear whether speed affects the process of audiovisual temporal integration in motion perception. Therefore, this study used a streaming-bouncing task (a bistable motion perception; SB task) combined with a simultaneous judgment task (SJ task) to explore the effect of speed on audiovisual temporal integration from implicit and explicit perspectives. The experiment had a within-subjects design, two speed conditions (fast/slow), eleven audiovisual conditions [stimulus onset asynchrony (SOA): 0 ms/ ± 60 ms/ ± 120 ms/ ± 180 ms/ ± 240 ms/ ± 300 ms], and a visual-only condition. A total of 30 subjects were recruited for the study. These participants completed the SB task and the SJ task successively. The results showed the following outcomes: (1) the optimal times needed to induce the "bouncing" illusion and maximum audiovisual bounce-inducing effect (ABE) magnitude were much earlier than that for the optimal time of audiovisual synchrony, (2) speed as a bottom-up factor could affect the proportion of "bouncing" perception in SB illusions but did not affect the ABE magnitude, (3) speed could also affect the ability of audiovisual temporal integration in motion perception, and the main manifestation was that the point of subjective simultaneity (PSS) in fast speed conditions was earlier than that of slow speed conditions in the SJ task and (4) the SB task and SJ task were not related. In conclusion, the time to complete the maximum audiovisual integration was different from the optimal time for synchrony perception; moreover, speed could affect audiovisual temporal integration in motion perception but only in explicit temporal tasks.

The Effects of Bilateral and Ipsilateral Auditory Stimuli on the Subcomponents of Visual Attention

Attention contains three functional network subcomponents of alerting, orienting, and executive control. The attention network test (ANT) is usually used to measure the efficiency of three attention subcomponents. Previous researches have focused on examining the unimodal attention with visual or auditory ANT paradigms. However, it is still unclear how an auditory stimulus influences the visual attention networks. This study investigated the effects of bilateral auditory stimuli (Experiment 1) and ipsilateral auditory stimulus (Experiment 2) on the visual attention subcomponents. We employed an ANT paradigm and manipulated the target modality types, including visual and audiovisual modalities. The participants were instructed to distinguish the direction of the central arrow surrounded by distractor arrows. In Experiment 1, we found that the simultaneous bilateral auditory stimuli reduced the efficiency of visual alerting and orienting, but had no significant effect on the efficiency of visual executive control. In Experiment 2, the ipsilateral auditory stimulus reduced the efficiency of visual executive control, but had no significant effect on the efficiency of visual alerting and orienting. We also observed a reduced relative multisensory response enhancement (rMRE) effect in cue condition relative to no cue condition (Experiment 1), and an increased rMRE effect in congruent condition compared with incongruent condition (Experiment 2). These results firstly provide evidence for the alerting, orienting and executive control effects in audiovisual condition. And the bilateral and ipsilateral auditory stimuli have different effects on the subcomponents of visual attention.

Modal-based attention modulates attentional blink

In the rapid serial visual presentation (RSVP) paradigm, response accuracy for the target decreases when it appears within a short time window (200~500 ms) after the previous target. This phenomenon is termed the attentional blink (AB). Although mechanisms of cross-modal processing that reduce the AB have been documented, researchers have not explored the differences across modal attentional conditions. In the present study, we used the RSVP paradigm to investigate the effect of auditory-driven visual target perceptual enhancement on the AB under modality-specific selective attention (Experiment 1) and bimodal-divided attention (Experiment 2). The results showed that cross-modal attentional enhancement was not moderated by stimulus salience. Moreover, the results also showed that accuracy was higher when the attended sound appeared simultaneously with the target. These results indicated that audiovisual enhancement reduced AB and that stronger attentional enhancement in the bimodal-divided attentional condition led to the disappearance of AB.

Updating the dual-mechanism model for cross-sensory attentional spreading: The influence of space-based visual selective attention

Selective attention to visual stimuli can spread cross‐modally to task‐irrelevant auditory stimuli through either the stimulus‐driven binding mechanism or the representation‐driven priming mechanism. The stimulus‐driven attentional spreading occurs whenever a task‐irrelevant sound is delivered simultaneously with a spatially attended visual stimulus, whereas the representation‐driven attentional spreading occurs only when the object representation of the sound is congruent with that of the to‐be‐attended visual object. The current study recorded event‐related potentials in a space‐selective visual object‐recognition task to examine the exact roles of space‐based visual selective attention in both the stimulus‐driven and representation‐driven cross‐modal attentional spreading, which remain controversial in the literature. Our results yielded that the representation‐driven auditory Nd component (200–400 ms after sound onset) did not differ according to whether the peripheral visual representations of audiovisual target objects were spatially attended or not, but was decreased when the auditory representations of target objects were presented alone. In contrast, the stimulus‐driven auditory Nd component (200–300 ms) was decreased but still prominent when the peripheral visual constituents of audiovisual nontarget objects were spatially unattended. These findings demonstrate not only that the representation‐driven attentional spreading is independent of space‐based visual selective attention and benefits in an all‐or‐nothing manner from object‐based visual selection for actually presented visual representations of target objects, but also that although the stimulus‐driven attentional spreading is modulated by space‐based visual selective attention, attending to visual modality per se is more likely to be the endogenous determinant of the stimulus‐driven attentional spreading.

Multisensory stimuli shift perceptual priors to facilitate rapid behavior

Multisensory stimuli speed behavioral responses, but the mechanisms subserving these effects remain disputed. Historically, the observation that multisensory reaction times (RTs) outpace models assuming independent sensory channels has been taken as evidence for multisensory integration (the "redundant target effect"; RTE). However, this interpretation has been challenged by alternative explanations based on stimulus sequence effects, RT variability, and/or negative correlations in unisensory processing. To clarify the mechanisms subserving the RTE, we collected RTs from 78 undergraduates in a multisensory simple RT task. Based on previous neurophysiological findings, we hypothesized that the RTE was unlikely to reflect these alternative mechanisms, and more likely reflected pre-potentiation of sensory responses through crossmodal phase-resetting. Contrary to accounts based on stimulus sequence effects, we found that preceding stimuli explained only 3-9% of the variance in apparent RTEs. Comparing three plausible evidence accumulator models, we found that multisensory RT distributions were best explained by increased sensory evidence at stimulus onset. Because crossmodal phase-resetting increases cortical excitability before sensory input arrives, these results are consistent with a mechanism based on pre-potentiation through phase-resetting. Mathematically, this model entails increasing the prior log-odds of stimulus presence, providing a potential link between neurophysiological, behavioral, and computational accounts of multisensory interactions.

Neurofeedback Modulation of the Sound-induced Flash Illusion Using Parietal Cortex Alpha Oscillations Reveals Dependency on Prior Multisensory Congruency

Spontaneous neural oscillations are key predictors of perceptual decisions to bind multisensory signals into a unified percept. Research links decreased alpha power in the posterior cortices to attention and audiovisual binding in the sound-induced flash illusion (SIFI) paradigm. This suggests that controlling alpha oscillations would be a way of controlling audiovisual binding. In the present feasibility study we used MEG-neurofeedback to train one group of subjects to increase left/right and another to increase right/left alpha power ratios in the parietal cortex. We tested for changes in audiovisual binding in a SIFI paradigm where flashes appeared in both hemifields. Results showed that the neurofeedback induced a significant asymmetry in alpha power for the left/right group, not seen for the right/left group. Corresponding asymmetry changes in audiovisual binding in illusion trials (with 2, 3, and 4 beeps paired with 1 flash) were not apparent. Exploratory analyses showed that neurofeedback training effects were present for illusion trials with the lowest numeric disparity (i.e., 2 beeps and 1 flash trials) only if the previous trial had high congruency (2 beeps and 2 flashes). Our data suggest that the relation between parietal alpha power (an index of attention) and its effect on audiovisual binding is dependent on the learned causal structure in the previous stimulus. The present results suggests that low alpha power biases observers towards audiovisual binding when they have learned that audiovisual signals originate from a common origin, consistent with a Bayesian causal inference account of multisensory perception.

Precision control for a flexible body representation

Adaptive body representation requires the continuous integration of multisensory inputs within a flexible 'body model' in the brain. The present review evaluates the idea that this flexibility is augmented by the contextual modulation of sensory processing 'top-down'; which can be described as precision control within predictive coding formulations of Bayesian inference. Specifically, I focus on the proposal that an attenuation of proprioception may facilitate the integration of conflicting visual and proprioceptive bodily cues. Firstly, I review empirical work suggesting that the processing of visual vs proprioceptive body position information can be contextualised 'top-down'; for instance, by adopting specific attentional task sets. Building up on this, I review research showing a similar contextualisation of visual vs proprioceptive information processing in the rubber hand illusion and in visuomotor adaptation. Together, the reviewed literature suggests that proprioception, despite its indisputable importance for body perception and action control, can be attenuated top-down (through precision control) to facilitate the contextual adaptation of the brain's body model to novel visual feedback.

Inhibition of Return Decreases Early Audiovisual Integration: An Event-Related Potential Study

Previous behavioral studies have found that inhibition of return decreases the audiovisual integration, while the underlying neural mechanisms are unknown. The current work utilized the high temporal resolution of event-related potentials (ERPs) to investigate how audiovisual integration would be modulated by inhibition of return. We employed the cue-target paradigm and manipulated the target type and cue validity. Participants were required to perform the task of detection of visual (V), auditory (A), or audiovisual (AV) targets shown in the identical (valid cue) or opposed (invalid cue) side to be the preceding exogenous cue. The neural activities between AV targets and the sum of the A and V targets were compared, and their differences were calculated to present the audiovisual integration effect in different cue validity conditions (valid, invalid). The ERPs results showed that a significant super-additive audiovisual integration effect was observed on the P70 (60∼90 ms, frontal-central) only under the invalid cue condition. The significant audiovisual integration effects were observed on the N1 or P2 components (N1, 120∼180 ms, frontal-central-parietal; P2, 200∼260 ms, frontal-central-parietal) in both valid cue as well as invalid cue condition. And there were no significant differences on the later components between invalid cue and valid cue. The result offers the first neural demonstration that inhibition of return modulates the early audiovisual integration process.

Maladaptive reorganization following SCI: The role of body representation and multisensory integration

In this review we focus on maladaptive brain reorganization after spinal cord injury (SCI), including the development of neuropathic pain, and its relationship with impairments in body representation and multisensory integration. We will discuss the implications of altered sensorimotor interactions after SCI with and without neuropathic pain and possible deficits in multisensory integration and body representation. Within this framework we will examine published research findings focused on the use of bodily illusions to manipulate multisensory body representation to induce analgesic effects in heterogeneous chronic pain populations and in SCI-related neuropathic pain. We propose that the development and intensification of neuropathic pain after SCI is partly dependent on brain reorganization associated with dysfunctional multisensory integration processes and distorted body representation. We conclude this review by suggesting future research avenues that may lead to a better understanding of the complex mechanisms underlying the sense of the body after SCI, with a focus on cortical changes.

Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance

Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.

Early Capture of Attention by Self-Face: Investigation Using a Temporal Order Judgment Task

Earlier work on self-face processing has reported a bias in the processing of self-face result in faster response to self-face in comparison to other familiar and unfamiliar faces (termed as self-face advantage or SFA). Even though most studies agree that the SFA occurs due to an attentional bias, there is little agreement regarding the stage at which it occurs. While a large number of studies show self-face influencing processing later at disengagement stage, early event-related potential components show differential activity for the self-face suggesting that SFA occurs early. We address this contradiction using a cueless temporal order judgment task that allows us to investigate early perceptual processing, while bias due to top-down expectation is controlled. A greater shift in point of subjective simultaneity for self-face would indicate a greater processing advantage at early perceptual stage. With help of two experiments, we show an early perceptual advantage for self-face, compared to both a friend's face and an unfamiliar face (Experiment 1). This advantage is present even when the effect of criterion shift is minimized (Experiment 2). Interestingly, the magnitude of advantage is similar for self-friend and self-unfamiliar pair. The evidence from the two experiments suggests early capture of attention as a likely reason for the SFA, which is present for the self-face but not for other familiar faces.

Endogenous Spatial Attention Modulates the Magnitude of the Colavita Visual Dominance Effect

The Colavita effect refers to the phenomenon wherein people tend to not respond to an auditory stimulus when a visual stimulus is simultaneously presented. Although previous studies have shown that endogenous modality attention influences the Colavita effect, whether the Colavita effect is influenced by endogenous spatial attention remains unknown. In the present study, we established endogenous spatial cues to investigate whether the size of the Colavita effect changes under visual or auditory cues. We measured three indexes to investigate the effect of endogenous spatial attention on the size of the Colavita effect. These three indexes were developed based on the following observations in bimodal trials: (a) The proportion of the "only vision" response was significantly higher than that of the "only audition" response; (b) the proportion of the "vision precedes audition" response was significantly higher than that of the "audition precedes vision" response; and (c) the reaction time difference of the "vision precedes audition" response was significantly higher than that of the "audition precedes vision" response. Our results showed that the Colavita effect was always influenced by endogenous spatial attention and that its size was larger at the cued location than at the uncued location; the cue modality (visual vs. auditory) had no effect on the size of the Colavita effect. Taken together, the present results shed light on how endogenous spatial attention affects the Colavita effect.

Attentional Modulation of Vision Versus Proprioception During Action

To control our actions efficiently, our brain represents our body based on a combination of visual and proprioceptive cues, weighted according to how (un)reliable—how precise—each respective modality is in a given context. However, perceptual experiments in other modalities suggest that the weights assigned to sensory cues are also modulated “top-down” by attention. Here, we asked whether during action, attention can likewise modulate the weights (i.e., precision) assigned to visual versus proprioceptive information about body position. Participants controlled a virtual hand (VH) via a data glove, matching either the VH or their (unseen) real hand (RH) movements to a target, and thus adopting a ``visual'' or ``proprioceptive'' attentional set, under varying levels of visuo-proprioceptive congruence and visibility. Functional magnetic resonance imaging (fMRI) revealed increased activation of the multisensory superior parietal lobe (SPL) during the VH task and increased activation of the secondary somatosensory cortex (S2) during the RH task. Dynamic causal modeling (DCM) showed that these activity changes were the result of selective, diametrical gain modulations in the primary visual cortex (V1) and the S2. These results suggest that endogenous attention can balance the gain of visual versus proprioceptive brain areas, thus contextualizing their influence on multisensory areas representing the body for action.

Cross-modal orienting of exogenous attention results in visual-cortical facilitation, not suppression

Attention may be oriented exogenously (i.e., involuntarily) to the location of salient stimuli, resulting in improved perception. However, it is unknown whether exogenous attention improves perception by facilitating processing of attended information, suppressing processing of unattended information, or both. To test this question, we measured behavioral performance and cue-elicited neural changes in the electroencephalogram as participants (N = 19) performed a task in which a spatially non-predictive auditory cue preceded a visual target. Critically, this cue was either presented at a peripheral target location or from the center of the screen, allowing us to isolate spatially specific attentional activity. We find that both behavior and attention-mediated changes in visual-cortical activity are enhanced at the location of a cue prior to the onset of a target, but that behavior and neural activity at an unattended target location is equivalent to that following a central cue that does not direct attention (i.e., baseline). These results suggest that exogenous attention operates via facilitation of information at an attended location.

Exogenous Bimodal Cues Attenuate Age-Related Audiovisual Integration

Previous studies have demonstrated that exogenous attention decreases audiovisual integration (AVI); however, whether the AVI is different when exogenous attention is elicited by bimodal and unimodal cues and its aging effect remain unclear. To clarify this matter, 20 older adults and 20 younger adults were recruited to conduct an auditory/visual discrimination task following bimodal audiovisual cues or unimodal auditory/visual cues. The results showed that the response to all stimulus types was faster in younger adults compared with older adults, and the response was faster when responding to audiovisual stimuli compared with auditory or visual stimuli. Analysis using the race model revealed that the AVI was lower in the exogenous-cue conditions compared with the no-cue condition for both older and younger adults. The AVI was observed in all exogenous-cue conditions for the younger adults (visual cue > auditory cue > audiovisual cue); however, for older adults, the AVI was only found in the visual-cue condition. In addition, the AVI was lower in older adults compared to younger adults under no- and visual-cue conditions. These results suggested that exogenous attention decreased the AVI, and the AVI was lower in exogenous attention elicited by bimodal-cue than by unimodal-cue conditions. In addition, the AVI was reduced for older adults compared with younger adults under exogenous attention.

Effects of Audiovisual Presentations on Visual Localization Errors: One or Several Multisensory Mechanisms?

The present study examines the extent to which temporal and spatial properties of sound modulate visual motion processing in spatial localization tasks. Participants were asked to locate the place at which a moving visual target unexpectedly vanished. Across different tasks, accompanying sounds were factorially varied within subjects as to their onset and offset times and/or positions relative to visual motion. Sound onset had no effect on the localization error. Sound offset was shown to modulate the perceived visual offset location, both for temporal and spatial disparities. This modulation did not conform to attraction toward the timing or location of the sounds but, demonstrably in the case of temporal disparities, to bimodal enhancement instead. Favorable indications to a contextual effect of audiovisual presentations on interspersed visual-only trials were also found. The short sound-leading offset asynchrony had equivalent benefits to audiovisual offset synchrony, suggestive of the involvement of early-level mechanisms, constrained by a temporal window, at these conditions. Yet, we tentatively hypothesize that the whole of the results and how they compare with previous studies requires the contribution of additional mechanisms, including learning-detection of auditory-visual associations and cross-sensory spread of endogenous attention.

Deficient auditory emotion processing but intact emotional multisensory integration in alexithymia

Alexithymia has been associated with emotion recognition deficits in both auditory and visual domains. Although emotions are inherently multimodal in daily life, little is known regarding abnormalities of emotional multisensory integration (eMSI) in relation to alexithymia. Here, we employed an emotional Stroop‐like audiovisual task while recording event‐related potentials (ERPs) in individuals with high alexithymia levels (HA) and low alexithymia levels (LA). During the task, participants had to indicate whether a voice was spoken in a sad or angry prosody while ignoring the simultaneously presented static face which could be either emotionally congruent or incongruent to the human voice. We found that HA performed worse and showed higher P2 amplitudes than LA independent of emotion congruency. Furthermore, difficulties in identifying and describing feelings were positively correlated with the P2 component, and P2 correlated negatively with behavioral performance. Bayesian statistics showed no group differences in eMSI and classical integration‐related ERP components (N1 and N2). Although individuals with alexithymia indeed showed deficits in auditory emotion recognition as indexed by decreased performance and higher P2 amplitudes, the present findings suggest an intact capacity to integrate emotional information from multiple channels in alexithymia. Our work provides valuable insights into the relationship between alexithymia and neuropsychological mechanisms of emotional multisensory integration.

Our behavioral and electrophysiological data provide substantial evidence for intact emotion multisensory integration in relation to alexithymia. With high ecological validity, these findings are of particular importance given that humans are constantly exposed to competing, complex audiovisual emotional information in social interaction contexts. Our work has important implications for the psychophysiology of alexithymia and emotional processing.

Trimodal processing of complex stimuli in inferior parietal cortex is modality-independent

In humans, multisensory mechanisms facilitate object processing through integration of sensory signals that match in their temporal and spatial occurrence as well as their meaning. The generalizability of such integration processes across different sensory modalities is, however, to date not well understood. As such, it remains unknown whether there are cerebral areas that process object-related signals independently of the specific senses from which they arise, and whether these areas show different response profiles depending on the number of sensory channels that carry information. To address these questions, we presented participants with dynamic stimuli that simultaneously emitted object-related sensory information via one, two, or three channels (sight, sound, smell) in the MR scanner. By comparing neural activation patterns between various integration processes differing in type and number of stimulated senses, we showed that the left inferior frontal gyrus and areas within the left inferior parietal cortex were engaged independently of the number and type of sensory input streams. Activation in these areas was enhanced during bimodal stimulation, compared to the sum of unimodal activations, and increased even further during trimodal stimulation. Taken together, our findings demonstrate that activation of the inferior parietal cortex during processing and integration of meaningful multisensory stimuli is both modality-independent and modulated by the number of available sensory modalities. This suggests that the processing demand placed on the parietal cortex increases with the number of sensory input streams carrying meaningful information, likely due to the increasing complexity of such stimuli.

Integration of vestibular and hindlimb inputs by vestibular nucleus neurons: multisensory influences on postural control

We recently demonstrated in decerebrate and conscious cat preparations that hindlimb somatosensory inputs converge with vestibular afferent input onto neurons in multiple central nervous system (CNS) locations that participate in balance control. Although it is known that head position and limb state modulate postural reflexes, presumably through vestibulospinal and reticulospinal pathways, the combined influence of the two inputs on the activity of neurons in these brainstem regions is unknown. In the present study, we evaluated the responses of vestibular nucleus (VN) neurons to vestibular and hindlimb stimuli delivered separately and together in conscious cats. We hypothesized that VN neuronal firing during activation of vestibular and limb proprioceptive inputs would be well fit by an additive model. Extracellular single-unit recordings were obtained from VN neurons. Sinusoidal whole body rotation in the roll plane was used as the search stimulus. Units responding to the search stimulus were tested for their responses to 10° ramp-and-hold roll body rotation, 60° extension hindlimb movement, and both movements delivered simultaneously. Composite response histograms were fit by a model of low- and high-pass filtered limb and body position signals using least squares nonlinear regression. We found that VN neuronal activity during combined vestibular and hindlimb proprioceptive stimulation in the conscious cat is well fit by a simple additive model for signals with similar temporal dynamics. The mean R² value for goodness of fit across all units was 0.74 ± 0.17. It is likely that VN neurons that exhibit these integrative properties participate in adjusting vestibulospinal outflow in response to limb state.NEW & NOTEWORTHY Vestibular nucleus neurons receive convergent information from hindlimb somatosensory inputs and vestibular inputs. In this study, extracellular single-unit recordings of vestibular nucleus neurons during conditions of passively applied limb movement, passive whole body rotations, and combined stimulation were well fit by an additive model. The integration of hindlimb somatosensory inputs with vestibular inputs at the first stage of vestibular processing suggests that vestibular nucleus neurons account for limb position in determining vestibulospinal responses to postural perturbations.