Stimulus value gates multisensory integration

The brain can develop conflicting multisensory principles to guide behavior

Midbrain multisensory neurons undergo a significant postnatal transition in how they process cross-modal (e.g. visual-auditory) signals. In early stages, signals derived from common events are processed competitively; however, at later stages they are processed cooperatively such that their salience is enhanced. This transition reflects adaptation to cross-modal configurations that are consistently experienced and become informative about which correspond to common events. Tested here was the assumption that overt behaviors follow a similar maturation. Cats were reared in omnidirectional sound thereby compromising the experience needed for this developmental process. Animals were then repeatedly exposed to different configurations of visual and auditory stimuli (e.g. spatiotemporally congruent or spatially disparate) that varied on each side of space and their behavior was assessed using a detection/localization task. Animals showed enhanced performance to stimuli consistent with the experience provided: congruent stimuli elicited enhanced behaviors where spatially congruent cross-modal experience was provided, and spatially disparate stimuli elicited enhanced behaviors where spatially disparate cross-modal experience was provided. Cross-modal configurations not consistent with experience did not enhance responses. The presumptive benefit of such flexibility in the multisensory developmental process is to sensitize neural circuits (and the behaviors they control) to the features of the environment in which they will function. These experiments reveal that these processes have a high degree of flexibility, such that two (conflicting) multisensory principles can be implemented by cross-modal experience on opposite sides of space even within the same animal.

Cross-modal exposure restores multisensory enhancement after hemianopia

Hemianopia is a common consequence of unilateral damage to visual cortex that manifests as a profound blindness in contralesional space. A noninvasive cross-modal (visual-auditory) exposure paradigm has been developed in an animal model to ameliorate this disorder. Repeated stimulation of a visual-auditory stimulus restores overt responses to visual stimuli in the blinded hemifield. It is believed to accomplish this by enhancing the visual sensitivity of circuits remaining after a lesion of visual cortex; in particular, circuits involving the multisensory neurons of the superior colliculus. Neurons in this midbrain structure are known to integrate spatiotemporally congruent visual and auditory signals to amplify their responses, which, in turn, enhances behavioral performance. Here we evaluated the relationship between the rehabilitation of hemianopia and this process of multisensory integration. Induction of hemianopia also eliminated multisensory enhancement in the blinded hemifield. Both vision and multisensory enhancement rapidly recovered with the rehabilitative cross-modal exposures. However, although both reached pre-lesion levels at similar rates, they did so with different spatial patterns. The results suggest that the capability for multisensory integration and enhancement is not a pre-requisite for visual recovery in hemianopia, and that the underlying mechanisms for recovery may be more complex than currently appreciated.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Task Instructions and the Need for Feedback Correction Influence the Contribution of Visual Errors to Reach Adaptation

Previous research has questioned whether motor adaptation is shaped by an optimal combination of multisensory error signals. Here, we expanded on this work by investigating how the use of visual and somatosensory error signals during online correction influences single-trial adaptation. To this end, we exposed participants to a random sequence of force-field perturbations and recorded their corrective responses as well as the after-effects exhibited during the subsequent unperturbed movement. In addition to the force perturbation, we artificially decreased or increased visual errors by multiplying hand deviations by a gain smaller or larger than one. Corrective responses to the force perturbation clearly scaled with the size of the visual error, but this scaling did not transfer one-to-one to motor adaptation and we observed no consistent interaction between limb and visual errors on adaptation. However, reducing visual errors during perturbation led to a small reduction of after-effects and this residual influence of visual feedback was eliminated when we instructed participants to control their hidden hand instead of the visual hand cursor. Taken together, our results demonstrate that task instructions and the need to correct for errors during perturbation are important factors to consider if we want to understand how the sensorimotor system uses and combines multimodal error signals to adapt movements.

Noise-rearing precludes the behavioral benefits of multisensory integration

Concordant visual-auditory stimuli enhance the responses of individual superior colliculus (SC) neurons. This neuronal capacity for "multisensory integration" is not innate: it is acquired only after substantial cross-modal (e.g. auditory-visual) experience. Masking transient auditory cues by raising animals in omnidirectional sound ("noise-rearing") precludes their ability to obtain this experience and the ability of the SC to construct a normal multisensory (auditory-visual) transform. SC responses to combinations of concordant visual-auditory stimuli are depressed, rather than enhanced. The present experiments examined the behavioral consequence of this rearing condition in a simple detection/localization task. In the first experiment, the auditory component of the concordant cross-modal pair was novel, and only the visual stimulus was a target. In the second experiment, both component stimuli were targets. Noise-reared animals failed to show multisensory performance benefits in either experiment. These results reveal a close parallel between behavior and single neuron physiology in the multisensory deficits that are induced when noise disrupts early visual-auditory experience.

Cross-modal facilitation of auditory discrimination in a frog

Stimulation in one sensory modality can affect perception in a separate modality, resulting in diverse effects including illusions in humans. This can also result in cross-modal facilitation, a process where sensory performance in one modality is improved by stimulation in another modality. For instance, a simple sound can improve performance in a visual task in both humans and cats. However, the range of contexts and underlying mechanisms that evoke such facilitation effects remain poorly understood. Here, we demonstrated cross-modal stimulation in wild-caught túngara frogs, a species with well-studied acoustic preferences in females. We first identified that a combined visual and seismic cue (vocal sac movement and water ripple) was behaviourally relevant for females choosing between two courtship calls in a phonotaxis assay. We then found that this combined cross-modal stimulus rescued a species-typical acoustic preference in the presence of background noise that otherwise abolished the preference. These results highlight how cross-modal stimulation can prime attention in receivers to improve performance during decision-making. With this, we provide the foundation for future work uncovering the processes and conditions that promote cross-modal facilitation effects.