Multisensory integration for orienting responses in humans requires the activation of the superior colliculus

Involvement of the superior colliculi in crossmodal correspondences

There is an increasing body of evidence suggesting that there are low-level perceptual processes involved in crossmodal correspondences. In this study, we investigate the involvement of the superior colliculi in three basic crossmodal correspondences: elevation/pitch, lightness/pitch, and size/pitch. Using a psychophysical design, we modulate visual input to the superior colliculus to test whether the superior colliculus is required for behavioural crossmodal congruency effects to manifest in an unspeeded multisensory discrimination task. In the elevation/pitch task, superior colliculus involvement is required for a behavioural elevation/pitch congruency effect to manifest in the task. In the lightness/pitch and size/pitch task, we observed a behavioural elevation/pitch congruency effect regardless of superior colliculus involvement. These results suggest that the elevation/pitch correspondence may be processed differently to other low-level crossmodal correspondences. The implications of a distributed model of crossmodal correspondence processing in the brain are discussed.

Multisensory stimuli shift perceptual priors to facilitate rapid behavior

Multisensory stimuli speed behavioral responses, but the mechanisms subserving these effects remain disputed. Historically, the observation that multisensory reaction times (RTs) outpace models assuming independent sensory channels has been taken as evidence for multisensory integration (the "redundant target effect"; RTE). However, this interpretation has been challenged by alternative explanations based on stimulus sequence effects, RT variability, and/or negative correlations in unisensory processing. To clarify the mechanisms subserving the RTE, we collected RTs from 78 undergraduates in a multisensory simple RT task. Based on previous neurophysiological findings, we hypothesized that the RTE was unlikely to reflect these alternative mechanisms, and more likely reflected pre-potentiation of sensory responses through crossmodal phase-resetting. Contrary to accounts based on stimulus sequence effects, we found that preceding stimuli explained only 3-9% of the variance in apparent RTEs. Comparing three plausible evidence accumulator models, we found that multisensory RT distributions were best explained by increased sensory evidence at stimulus onset. Because crossmodal phase-resetting increases cortical excitability before sensory input arrives, these results are consistent with a mechanism based on pre-potentiation through phase-resetting. Mathematically, this model entails increasing the prior log-odds of stimulus presence, providing a potential link between neurophysiological, behavioral, and computational accounts of multisensory interactions.

Multisensory signals inhibit pupillary light reflex: Evidence from pupil oscillation

Multisensory integration, which enhances stimulus saliency at the early stage of the processing hierarchy, has been recently shown to produce a larger pupil size than its unisensory constituents. Theoretically, any modulation on pupil size ought to be associated with the sympathetic and parasympathetic pathways that are sensitive to light. But it remains poorly understood how the pupillary light reflex is changed in a multisensory context. The present study evoked an oscillation of the pupillary light reflex by periodically changing the luminance of a visual stimulus at 1.25 Hz. It was found that such induced pupil size oscillation was substantially attenuated when the bright but not the dark phase of the visual flicker was periodically and synchronously presented with a burst of tones. This inhibition effect persisted when the visual flicker was task-irrelevant and out of attentional focus, but disappeared when the visual flicker was moved from the central field to the periphery. These findings not only offer a comprehensive characterization of the multisensory impact on pupil response to light, but also provide valuable clues about the individual contributions of the sympathetic and parasympathetic pathways to multisensory modulation of pupil size.

Shaping the visual system: cortical and subcortical plasticity in the intact and the lesioned brain

Visual system is endowed with an incredibly complex organization composed of multiple visual pathway affording both hierarchical and parallel processing. Even if most of the visual information is conveyed by the retina to the lateral geniculate nucleus of the thalamus and then to primary visual cortex, a wealth of alternative subcortical pathways is present. This complex organization is experience dependent and retains plastic properties throughout the lifespan enabling the system with a continuous update of its functions in response to variable external needs. Changes can be induced by several factors including learning and experience but can also be promoted by the use non-invasive brain stimulation techniques. Furthermore, besides the astonishing ability of our visual system to spontaneously reorganize after injuries, we now know that the exposure to specific rehabilitative training can produce not only important functional modifications but also long-lasting changes within cortical and subcortical structures. The present review aims to update and address the current state of the art on these topics gathering studies that reported relevant modifications of visual functioning together with plastic changes within cortical and subcortical structures both in the healthy and in the lesioned visual system.

Using superior colliculus principles of multisensory integration to reverse hemianopia

The diversity of our senses conveys many advantages; it enables them to compensate for one another when needed, and the information they provide about a common event can be integrated to facilitate its processing and, ultimately, adaptive responses. These cooperative interactions are produced by multisensory neurons. A well-studied model in this context is the multisensory neuron in the output layers of the superior colliculus (SC). These neurons integrate and amplify their cross-modal (e.g., visual-auditory) inputs, thereby enhancing the physiological salience of the initiating event and the probability that it will elicit SC-mediated detection, localization, and orientation behavior. Repeated experience with the same visual-auditory stimulus can also increase the neuron's sensitivity to these individual inputs. This observation raised the possibility that such plasticity could be engaged to restore visual responsiveness when compromised. For example, unilateral lesions of visual cortex compromise the visual responsiveness of neurons in the multisensory output layers of the ipsilesional SC and produces profound contralesional blindness (hemianopia). The possibility that multisensory plasticity could restore the visual responses of these neurons, and reverse blindness, was tested in the cat model of hemianopia. Hemianopic subjects were repeatedly presented with spatiotemporally congruent visual-auditory stimulus pairs in the blinded hemifield on a daily or weekly basis. After several weeks of this multisensory exposure paradigm, visual responsiveness was restored in SC neurons and behavioral responses were elicited by visual stimuli in the previously blind hemifield. The constraints on the effectiveness of this procedure proved to be the same as those constraining SC multisensory plasticity: whereas repetitions of a congruent visual-auditory stimulus was highly effective, neither exposure to its individual component stimuli, nor to these stimuli in non-congruent configurations was effective. The restored visual responsiveness proved to be robust, highly competitive with that in the intact hemifield, and sufficient to support visual discrimination.

Improving Human–Computer Interface Design through Application of Basic Research on Audiovisual Integration and Amplitude Envelope

Quality care for patients requires effective communication amongst medical teams. Increasingly, communication is required not only between team members themselves, but between members and the medical devices monitoring and managing patient well-being. Most human–computer interfaces use either auditory or visual displays, and despite significant experimentation, they still elicit well-documented concerns. Curiously, few interfaces explore the benefits of multimodal communication, despite extensive documentation of the brain’s sensitivity to multimodal signals. New approaches built on insights from basic audiovisual integration research hold the potential to improve future human–computer interfaces. In particular, recent discoveries regarding the acoustic property of amplitude envelope illustrate that it can enhance audiovisual integration while also lowering annoyance. Here, we share key insights from recent research with the potential to inform applications related to human–computer interface design. Ultimately, this could lead to a cost-effective way to improve communication in medical contexts—with signification implications for both human health and the burgeoning medical device industry.

Responding to sounds from unseen locations: crossmodal attentional orienting in response to sounds presented from the rear

To date, most of the research on spatial attention has focused on probing people's responses to stimuli presented in frontal space. That is, few researchers have attempted to assess what happens in the space that is currently unseen (essentially rear space). In a sense, then, 'out of sight' is, very much, 'out of mind'. In this review, we highlight what is presently known about the perception and processing of sensory stimuli (focusing on sounds) whose source is not currently visible. We briefly summarize known differences in the localizability of sounds presented from different locations in 3D space, and discuss the consequences for the crossmodal attentional and multisensory perceptual interactions taking place in various regions of space. The latest research now clearly shows that the kinds of crossmodal interactions that take place in rear space are very often different in kind from those that have been documented in frontal space. Developing a better understanding of how people respond to unseen sound sources in naturalistic environments by integrating findings emerging from multiple fields of research will likely lead to the design of better warning signals in the future. This review highlights the need for neuroscientists interested in spatial attention to spend more time researching what happens (in terms of the covert and overt crossmodal orienting of attention) in rear space.

Compensatory Recovery after Multisensory Stimulation in Hemianopic Patients: Behavioral and Neurophysiological Components

Lateralized post-chiasmatic lesions of the primary visual pathway result in loss of visual perception in the field retinotopically corresponding to the damaged cortical area. However, patients with visual field defects have shown enhanced detection and localization of multisensory audio-visual pairs presented in the blind field. This preserved multisensory integrative ability (i.e., crossmodal blindsight) seems to be subserved by the spared retino-colliculo-dorsal pathway. According to this view, audio-visual integrative mechanisms could be used to increase the functionality of the spared circuit and, as a consequence, might represent an important tool for the rehabilitation of visual field defects. The present study tested this hypothesis, investigating whether exposure to systematic multisensory audio-visual stimulation could induce long-lasting improvements in the visual performance of patients with visual field defects. A group of 10 patients with chronic visual field defects were exposed to audio-visual training for 4 h daily, over a period of 2 weeks. Behavioral, oculomotor and electroencephalography (EEG) measures were recorded during several visual tasks before and after audio-visual training. After audio-visual training, improvements in visual search abilities, visual detection, self-perceived disability in daily life activities and oculomotor parameters were found, suggesting the implementation of more effective visual exploration strategies. At the electrophysiological level, after training, patients showed a significant reduction of the P3 amplitude in response to stimuli presented in the intact field, reflecting a reduction in attentional resources allocated to the intact field, which might co-occur with a shift of spatial attention towards the blind field. More interestingly, both the behavioral improvements and the electrophysiological changes observed after training were found to be stable at a follow-up session (on average, 8 months after training), suggesting long-term effects of multisensory audio-visual training. These long-lasting effects seem to be subserved by the activation of the spared retino-colliculo-dorsal pathway, which promotes orienting responses towards the blind field, able to both compensate for the visual field loss and concurrently attenuate visual attention towards the intact field. These results add to previous findings the knowledge that audio-visual multisensory stimulation promote long-term plastic changes in hemianopics, resulting in stable and long-lasting ameliorations in behavioral and electrophysiological measures.

Audio-visual multisensory training enhances visual processing of motion stimuli in healthy participants: an electrophysiological study

Evidence from electrophysiological and imaging studies suggests that audio-visual (AV) stimuli presented in spatial coincidence enhance activity in the subcortical colliculo-dorsal extrastriate pathway. To test whether repetitive AV stimulation might specifically activate this neural circuit underlying multisensory integrative processes, electroencephalographic data were recorded before and after 2 h of AV training, during the execution of two lateralized visual tasks: a motion discrimination task, relying on activity in the colliculo-dorsal MT pathway, and an orientation discrimination task, relying on activity in the striate and early ventral extrastriate cortices. During training, participants were asked to detect and perform a saccade towards AV stimuli that were disproportionally allocated to one hemifield (the trained hemifield). Half of the participants underwent a training in which AV stimuli were presented in spatial coincidence, while the remaining half underwent a training in which AV stimuli were presented in spatial disparity (32°). Participants who received AV training with stimuli in spatial coincidence had a post-training enhancement of the anterior N1 component in the motion discrimination task, but only in response to stimuli presented in the trained hemifield. However, no effect was found in the orientation discrimination task. In contrast, participants who received AV training with stimuli in spatial disparity showed no effects on either task. The observed N1 enhancement might reflect enhanced discrimination for motion stimuli, probably due to increased activity in the colliculo-dorsal MT pathway induced by multisensory training.

Visual rehabilitation: visual scanning, multisensory stimulation and vision restoration trainings

Neuropsychological training methods of visual rehabilitation for homonymous vision loss caused by postchiasmatic damage fall into two fundamental paradigms: “compensation” and “restoration”. Existing methods can be classified into three groups: Visual Scanning Training (VST), Audio-Visual Scanning Training (AViST) and Vision Restoration Training (VRT). VST and AViST aim at compensating vision loss by training eye scanning movements, whereas VRT aims at improving lost vision by activating residual visual functions by training light detection and discrimination of visual stimuli. This review discusses the rationale underlying these paradigms and summarizes the available evidence with respect to treatment efficacy. The issues raised in our review should help guide clinical care and stimulate new ideas for future research uncovering the underlying neural correlates of the different treatment paradigms. We propose that both local “within-system” interactions (i.e., relying on plasticity within peri-lesional spared tissue) and changes in more global “between-system” networks (i.e., recruiting alternative visual pathways) contribute to both vision restoration and compensatory rehabilitation, which ultimately have implications for the rehabilitation of cognitive functions.

Conditioned sounds enhance visual processing

This psychophysics study investigated whether prior auditory conditioning influences how a sound interacts with visual perception. In the conditioning phase, subjects were presented with three pure tones ( =  conditioned stimuli, CS) that were paired with positive, negative or neutral unconditioned stimuli. As unconditioned reinforcers we employed pictures (highly pleasant, unpleasant and neutral) or monetary outcomes (+50 euro cents, −50 cents, 0 cents). In the subsequent visual selective attention paradigm, subjects were presented with near-threshold Gabors displayed in their left or right hemifield. Critically, the Gabors were presented in synchrony with one of the conditioned sounds. Subjects discriminated whether the Gabors were presented in their left or right hemifields. Participants determined the location more accurately when the Gabors were presented in synchrony with positive relative to neutral sounds irrespective of reinforcer type. Thus, previously rewarded relative to neutral sounds increased the bottom-up salience of the visual Gabors. Our results are the first demonstration that prior auditory conditioning is a potent mechanism to modulate the effect of sounds on visual perception.

S-cone Visual Stimuli Activate Superior Colliculus Neurons in Old World Monkeys: Implications for Understanding Blindsight

The superior colliculus (SC) is thought to be unresponsive to stimuli that activate only short wavelength-sensitive cones (S-cones) in the retina. The apparent lack of S-cone input to the SC was recognized by Sumner et al. [Sumner, P., Adamjee, T., & Mollon, J. D. Signals invisible to the collicular and magnocellular pathways can capture visual attention. Current Biology, 12, 1312–1316, 2002] as an opportunity to test SC function. The idea is that visual behavior dependent on the SC should be impaired when S-cone stimuli are used because they are invisible to the SC. The SC plays a critical role in blindsight. If the SC is insensitive to S-cone stimuli blindsight behavior should be impaired when S-cone stimuli are used. Many clinical and behavioral studies have been based on the assumption that S-cone-specific stimuli do not activate neurons in the SC. Our goal was to test whether single neurons in macaque SC respond to stimuli that activate only S-cones. Stimuli were calibrated psychophysically in each animal and at each individual spatial location used in experimental testing [Hall, N. J., & Colby, C. L. Psychophysical definition of S-cone stimuli in the macaque. Journal of Vision, 13, 2013]. We recorded from 178 visually responsive neurons in two awake, behaving rhesus monkeys. Contrary to the prevailing view, we found that nearly all visual SC neurons can be activated by S-cone-specific visual stimuli. Most of these neurons were sensitive to the degree of S-cone contrast. Of 178 visual SC neurons, 155 (87%) had stronger responses to a high than to a low S-cone contrast. Many of these neurons' responses (56/178 or 31%) significantly distinguished between the high and low S-cone contrast stimuli. The latency and amplitude of responses depended on S-cone contrast. These findings indicate that stimuli that activate only S-cones cannot be used to diagnose collicular mediation.

Color selectivity of the spatial congruency effect: evidence from the focused attention paradigm

The multisensory response enhancement (MRE), occurring when the response to a visual target integrated with a spatially congruent sound is stronger than the response to the visual target alone, is believed to be mediated by the superior colliculus (SC) (Stein & Meredith, 1993). Here, we used a focused attention paradigm to show that the spatial congruency effect occurs with red (SC-effective) but not blue (SC-ineffective) visual stimuli, when presented with spatially congruent sounds. To isolate the chromatic component of SC-ineffective targets and to demonstrate the selectivity of the spatial congruency effect we used the random luminance modulation technique (Experiment 1) and the tritanopic technique (Experiment 2). Our results indicate that the spatial congruency effect does not require the distribution of attention over different sensory modalities and provide correlational evidence that the SC mediates the effect.

Brain deactivation in the outperformance in bimodal tasks: an FMRI study

While it is known that some individuals can effectively perform two tasks simultaneously, other individuals cannot. How the brain deals with performing simultaneous tasks remains unclear. In the present study, we aimed to assess which brain areas corresponded to various phenomena in task performance. Nineteen subjects were requested to sequentially perform three blocks of tasks, including two unimodal tasks and one bimodal task. The unimodal tasks measured either visual feature binding or auditory pitch comparison, while the bimodal task required performance of the two tasks simultaneously. The functional magnetic resonance imaging (fMRI) results are compatible with previous studies showing that distinct brain areas, such as the visual cortices, frontal eye field (FEF), lateral parietal lobe (BA7), and medial and inferior frontal lobe, are involved in processing of visual unimodal tasks. In addition, the temporal lobes and Brodmann area 43 (BA43) were involved in processing of auditory unimodal tasks. These results lend support to concepts of modality-specific attention. Compared to the unimodal tasks, bimodal tasks required activation of additional brain areas. Furthermore, while deactivated brain areas were related to good performance in the bimodal task, these areas were not deactivated where the subject performed well in only one of the two simultaneous tasks. These results indicate that efficient information processing does not require some brain areas to be overly active; rather, the specific brain areas need to be relatively deactivated to remain alert and perform well on two tasks simultaneously. Meanwhile, it can also offer a neural basis for biofeedback in training courses, such as courses in how to perform multiple tasks simultaneously.

How can audiovisual pathways enhance the temporal resolution of time-compressed speech in blind subjects?

In blind people, the visual channel cannot assist face-to-face communication via lipreading or visual prosody. Nevertheless, the visual system may enhance the evaluation of auditory information due to its cross-links to (1) the auditory system, (2) supramodal representations, and (3) frontal action-related areas. Apart from feedback or top-down support of, for example, the processing of spatial or phonological representations, experimental data have shown that the visual system can impact auditory perception at more basic computational stages such as temporal signal resolution. For example, blind as compared to sighted subjects are more resistant against backward masking, and this ability appears to be associated with activity in visual cortex. Regarding the comprehension of continuous speech, blind subjects can learn to use accelerated text-to-speech systems for "reading" texts at ultra-fast speaking rates (>16 syllables/s), exceeding by far the normal range of 6 syllables/s. A functional magnetic resonance imaging study has shown that this ability, among other brain regions, significantly covaries with BOLD responses in bilateral pulvinar, right visual cortex, and left supplementary motor area. Furthermore, magnetoencephalographic measurements revealed a particular component in right occipital cortex phase-locked to the syllable onsets of accelerated speech. In sighted people, the "bottleneck" for understanding time-compressed speech seems related to higher demands for buffering phonological material and is, presumably, linked to frontal brain structures. On the other hand, the neurophysiological correlates of functions overcoming this bottleneck, seem to depend upon early visual cortex activity. The present Hypothesis and Theory paper outlines a model that aims at binding these data together, based on early cross-modal pathways that are already known from various audiovisual experiments on cross-modal adjustments during space, time, and object recognition.

Eccentricity dependent auditory enhancement of visual stimulus detection but not discrimination

Sensory perception is enhanced by the complementary information provided by our different sensory modalities and even apparently task irrelevant stimuli in one modality can facilitate performance in another. While perception in general comprises both, the detection of sensory objects as well as their discrimination and recognition, most studies on audio-visual interactions have focused on either of these aspects. However, previous evidence, neuroanatomical projections between early sensory cortices and computational mechanisms suggest that sounds might differentially affect visual detection and discrimination and differentially at central and peripheral retinal locations. We performed an experiment to directly test this by probing the enhancement of visual detection and discrimination by auxiliary sounds at different visual eccentricities and within the same subjects. Specifically, we quantified the enhancement provided by sounds that reduce the overall uncertainty about the visual stimulus beyond basic multisensory co-stimulation. This revealed a general trend for stronger enhancement at peripheral locations in both tasks, but a statistically significant effect only for detection and only at peripheral locations. Overall this suggests that there are topographic differences in the auditory facilitation of basic visual processes and that these may differentially affect basic aspects of visual recognition.

Development of cortical influences on superior colliculus multisensory neurons: effects of dark-rearing

Rearing cats from birth to adulthood in darkness prevents neurons in the superior colliculus (SC) from developing the capability to integrate visual and non-visual (e.g. visual-auditory) inputs. Presumably, this developmental anomaly is due to a lack of experience with the combination of those cues, which is essential to form associative links between them. The visual-auditory multisensory integration capacity of SC neurons has also been shown to depend on the functional integrity of converging visual and auditory inputs from the ipsilateral association cortex. Disrupting these cortico-collicular projections at any stage of life results in a pattern of outcomes similar to those found after dark-rearing; SC neurons respond to stimuli in both sensory modalities, but cannot integrate the information they provide. Thus, it is possible that dark-rearing compromises the development of these descending tecto-petal connections and the essential influences they convey. However, the results of the present experiments, using cortical deactivation to assess the presence of cortico-collicular influences, demonstrate that dark-rearing does not prevent the association cortex from developing robust influences over SC multisensory responses. In fact, dark-rearing may increase their potency over that observed in normally-reared animals. Nevertheless, their influences are still insufficient to support SC multisensory integration. It appears that cross-modal experience shapes the cortical influence to selectively enhance responses to cross-modal stimulus combinations that are likely to be derived from the same event. In the absence of this experience, the cortex develops an indiscriminate excitatory influence over its multisensory SC target neurons.

Sound-driven enhancement of vision: disentangling detection-level from decision-level contributions

Cross-modal enhancement can be mediated both by higher-order effects due to attention and decision making and by detection-level stimulus-driven interactions. However, the contribution of each of these sources to behavioral improvements has not been conclusively determined and quantified separately. Here, we apply psychophysical analysis based on Piéron functions in order to separate stimulus-dependent changes from those accounted by decision-level contributions. Participants performed a simple visual speeded detection task on Gabor patches of different spatial frequencies and contrast values, presented with and without accompanying sounds. On one hand, we identified an additive cross-modal improvement in mean reaction times across all types of visual stimuli that would be well explained by interactions not strictly based on stimulus-driven modulations (e.g., due to reduction of temporal uncertainty and motor times). On the other hand, we singled out an audio-visual benefit that strongly depended on stimulus features such as frequency and contrast. This particular enhancement was selective to low-visual spatial frequency stimuli, optimized for magnocellular sensitivity. We therefore conclude that interactions at detection stages and at decisional processes in response selection that contribute to audio-visual enhancement can be separated online and express on partly different aspects of visual processing.

Multisensory plasticity in adulthood: cross-modal experience enhances neuronal excitability and exposes silent inputs

Multisensory superior colliculus neurons in cats were found to retain substantial plasticity to short-term, site-specific experience with cross-modal stimuli well into adulthood. Following cross-modal exposure trials, these neurons substantially increased their sensitivity to the cross-modal stimulus configuration as well as to its individual component stimuli. In many cases, the exposure experience also revealed a previously ineffective or "silent" input channel, rendering it overtly responsive. These experience-induced changes required relatively few exposure trials and could be retained for more than 1 h. However, their induction was generally restricted to experience with cross-modal stimuli. Only rarely were they induced by exposure to a modality-specific stimulus and were never induced by stimulating a previously ineffective input channel. This short-term plasticity likely provides substantial benefits to the organism in dealing with ongoing and sequential events that take place at a given location in space and may reflect the ability of multisensory superior colliculus neurons to rapidly alter their response properties to accommodate to changes in environmental challenges and event probabilities.

Neglect: a multisensory deficit?

Neglect is a neurological syndrome characterised by a lack of conscious perception of events localised in the contralesional side of space. Here, we consider the possible multisensory nature of this disorder, critically reviewing the literature devoted to multisensory manifestations and processing in neglect. Although its most striking manifestations have been observed in the visual domain, a number of studies demonstrate that neglect can affect virtually any sensory modality, in particular touch and audition. Furthermore, a few recent studies have reported a correlation in severity between visual and non-visual neglect-related deficits evaluated in the same patients, providing some preliminary support for a multisensory conception of neglect. Sensory stimulation and sensorimotor adaptation techniques, aimed at alleviating neglect, have also been shown to affect several sensory modalities, including some that were not directly affected by the intervention. Finally, in some cases neglect can bias multisensory interactions known to occur in healthy individuals, leading to abnormal behaviour or uncovering multisensory compensation mechanisms. This evidence, together with neurophysiological and neuroimaging data revealing the multisensory role played by the areas that are most commonly damaged in neglect patients, seems to speak in favour of neglect as a multisensory disorder. However, since most previous studies were not conducted with the specific purpose of systematically investigating the multisensory nature of neglect, we conclude that more research is needed to appropriately assess this question, and suggest some methodological guidelines that we hope will help clarify this issue. At present, the conception of neglect as a multisensory disorder remains a promising working hypothesis that may help define the pathophysiology of this syndrome.

Multisensory perceptual learning reshapes both fast and slow mechanisms of crossmodal processing

Previous research has shown that sounds facilitate perception of visual patterns appearing immediately after the sound but impair perception of patterns appearing after some delay. Here we examined the spatial gradient of the fast crossmodal facilitation effect and the slow inhibition effect in order to test whether they reflect separate mechanisms. We found that crossmodal facilitation is only observed at visual field locations overlapping with the sound, whereas crossmodal inhibition affects the whole hemifield. Furthermore, we tested whether multisensory perceptual learning with misaligned audio-visual stimuli reshapes crossmodal facilitation and inhibition. We found that training shifts crossmodal facilitation towards the trained location without changing its range. By contrast, training narrows the range of inhibition without shifting its position. Our results suggest that crossmodal facilitation and inhibition reflect separate mechanisms that can both be reshaped by multisensory experience even in adult humans. Multisensory links seem to be more plastic than previously thought.

Independent mechanisms for ventriloquism and multisensory integration as revealed by theta-burst stimulation

The visual and auditory systems often concur to create a unified perceptual experience and to determine the localization of objects in the external world. Co-occurring auditory and visual stimuli in spatial coincidence are known to enhance performance of auditory localization due to the integration of stimuli from different sensory channels (i.e. multisensory integration). However, auditory localization of audiovisual stimuli presented at spatial disparity might also induce a mislocalization of the sound towards the visual stimulus (i.e. ventriloquism effect). Using repetitive transcranial magnetic stimulation we tested the role of right temporoparietal (rTPC), right occipital (rOC) and right posterior parietal (rPPC) cortex in an auditory localization task in which indices of ventriloquism and multisensory integration were computed. We found that suppression of rTPC excitability by means of continuous theta-burst stimulation (cTBS) reduced multisensory integration. No similar effect was found for cTBS over rOC. Moreover, inhibition of rOC, but not of rTPC, suppressed the visual bias in the contralateral hemifield. In contrast, cTBS over rPPC did not produce any modulation of ventriloquism or integrative effects. The double dissociation found in the present study suggests that ventriloquism and audiovisual multisensory integration are functionally independent phenomena and may be underpinned by partially different neural circuits.

Collicular vision guides nonconscious behavior

Following destruction or deafferentation of primary visual cortex (area V1, striate cortex), clinical blindness ensues, but residual visual functions may, nevertheless, persist without perceptual consciousness (a condition termed blindsight). The study of patients with such lesions thus offers a unique opportunity to investigate what visual capacities are mediated by the extrastriate pathways that bypass V1. Here we provide evidence for a crucial role of the collicular-extrastriate pathway in nonconscious visuomotor integration by showing that, in the absence of V1, the superior colliculus (SC) is essential to translate visual signals that cannot be consciously perceived into motor outputs. We found that a gray stimulus presented in the blind field of a patient with unilateral V1 loss, although not consciously seen, can influence his behavioral and pupillary responses to consciously perceived stimuli in the intact field (implicit bilateral summation). Notably, this effect was accompanied by selective activations in the SC and in occipito-temporal extrastriate areas. However, when instead of gray stimuli we presented purple stimuli, which predominantly draw on S-cones and are thus invisible to the SC, any evidence of implicit visuomotor integration disappeared and activations in the SC dropped significantly. The present findings show that the SC acts as an interface between sensory and motor processing in the human brain, thereby providing a contribution to visually guided behavior that may remain functionally and anatomically segregated from the geniculo-striate pathway and entirely outside conscious visual experience.

Sounds can affect visual perception mediated primarily by the parvocellular pathway

We investigated the effect of auditory–visual sensory integration on visual tasks that were predominantly dependent on parvocellular processing. These tasks were (i) detecting metacontrast-masked targets and (ii) discriminating orientation differences between high spatial frequency Gabor patch stimuli. Sounds that contained no information relevant to either task were presented before, synchronized with, or after the visual targets, and the results were compared to conditions with no sound. Both tasks used a two-alternative forced choice technique. For detecting metacontrast-masked targets, one interval contained the visual target and both (or neither) intervals contained a sound. Sound–target synchrony within 50 ms lowered luminance thresholds for detecting the presence of a target compared to when no sound occurred or when sound onset preceded target onset. Threshold angles for discriminating the orientation of a Gabor patch consistently increased in the presence of a sound. These results are compatible with sound-induced activity in the parvocellular visual pathway increasing the visibility of flashed targets and hindering orientation discrimination.

Orienting of spatial attention and the interplay between the senses

Many everyday situations require combining complex sensory signals about the external world with ongoing goals and expectations. Here I examine the role of attention in this process and consider the underlying neural substrates. First, mechanisms of spatial attention in the visual modality are reviewed, emphasising the involvement of fronto-parietal cortex. Spatial attention takes into account endogenous factors, e.g., information about behavioural relevance, as well as signals arising from the external world (stimulus-driven control). Stimulus-driven control is thought to take place automatically and independently from endogenous factors. However, recent findings demonstrate that endogenous and stimulus-driven mechanisms co-operate, jointly contributing for the selection of the relevant spatial location. Next, I will turn to studies of multisensory spatial attention. These have shown that attention control in fronto-parietal cortex operates supramodally. Supramodal control exerts top-down influences onto sensory-specific areas, enhancing the processing of stimuli at the attended location irrespective of modality. Unlike unimodal visual attention, but in line with traditional views of multisensory integration, multisensory attention can operate in a fully automatic manner regardless of relevance and task-set. I discuss these findings in relation to functional/anatomical pathways that may mediate multisensory attention control, highlighting possible links between spatial attention and multisensory integration of space.

The differing impact of multisensory and unisensory integration on behavior

Pooling and synthesizing signals across different senses often enhances responses to the event from which they are derived. Here, we examine whether multisensory response enhancements are attributable to a redundant target effect (two stimuli rather than one) or if there is some special quality inherent in the combination of cues from different senses. To test these possibilities, the performance of animals in localizing and detecting spatiotemporally concordant visual and auditory stimuli was examined when these stimuli were presented individually (visual or auditory) or in cross-modal (visual-auditory) and within-modal (visual-visual, auditory-auditory) combinations. Performance enhancements proved to be far greater for combinations of cross-modal than within-modal stimuli and support the idea that the behavioral products derived from multisensory integration are not attributable to simple target redundancy. One likely explanation is that whereas cross-modal signals offer statistically independent samples of the environment, within-modal signals can exhibit substantial covariance, and consequently multisensory integration can yield more substantial error reduction than unisensory integration.

Audio-visual stimulation improves oculomotor patterns in patients with hemianopia

Patients with visual field disorders often exhibit impairments in visual exploration and a typical defective oculomotor scanning behaviour. Recent evidence [Bolognini, N., Rasi, F., Coccia, M., & Làdavas, E. (2005b). Visual search improvement in hemianopic patients after audio-visual stimulation. Brain, 128, 2830-2842] suggests that systematic audio-visual stimulation of the blind hemifield can improve accuracy and search times in visual exploration, probably due to the stimulation of Superior Colliculus (SC), an important multisensory structure involved in both the initiation and execution of saccades. The aim of the present study is to verify this hypothesis by studying the effects of multisensory training on oculomotor scanning behaviour. Oculomotor responses during a visual search task and a reading task were studied before and after visual (control) or audio-visual (experimental) training, in a group of 12 patients with chronic visual field defects and 12 controls subjects. Eye movements were recorded using an infra-red technique which measured a range of spatial and temporal variables. Prior to treatment, patients' performance was significantly different from that of controls in relation to fixations and saccade parameters; after Audio-Visual Training, all patients reported an improvement in ocular exploration characterized by fewer fixations and refixations, quicker and larger saccades, and reduced scanpath length. Overall, these improvements led to a reduction of total exploration time. Similarly, reading parameters were significantly affected by the training, with respect to specific impairments observed in both left- and right-hemianopia readers. Our findings provide evidence that Audio-Visual Training, by stimulating the SC, may induce a more organized pattern of visual exploration due to an implementation of efficient oculomotor strategies. Interestingly, the improvement was found to be stable at a 1 year follow-up control session, indicating a long-term persistence of treatment effects on the oculomotor system.

A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus

Sensory integration is a characteristic feature of superior colliculus (SC) neurons. A recent neural network model of single-neuron integration derived a set of basic biological constraints sufficient to replicate a number of physiological findings pertaining to multisensory responses. The present study examined the accuracy of this model in predicting the responses of SC neurons to pairs of visual stimuli placed within their receptive fields. The accuracy of this model was compared to that of three other computational models (additive, averaging and maximum operator) previously used to fit these data. Each neuron's behavior was assessed by examining its mean responses to the component stimuli individually and together, and each model's performance was assessed to determine how close its prediction came to the actual mean response of each neuron and the magnitude of its predicted residual error. Predictions from the additive model significantly overshot the actual responses of SC neurons and predictions from the averaging model significantly undershot them. Only the predictions of the maximum operator and neural network model were not significantly different from the actual responses. However, the neural network model outperformed even the maximum operator model in predicting the responses of these neurons. The neural network model is derived from a larger model that also has substantial predictive power in multisensory integration, and provides a single computational vehicle for assessing the responses of SC neurons to different combinations of cross-modal and within-modal stimuli of different efficacies.