Sensory and multisensory responses in the newborn monkey superior colliculus

The brain can develop conflicting multisensory principles to guide behavior

Midbrain multisensory neurons undergo a significant postnatal transition in how they process cross-modal (e.g. visual-auditory) signals. In early stages, signals derived from common events are processed competitively; however, at later stages they are processed cooperatively such that their salience is enhanced. This transition reflects adaptation to cross-modal configurations that are consistently experienced and become informative about which correspond to common events. Tested here was the assumption that overt behaviors follow a similar maturation. Cats were reared in omnidirectional sound thereby compromising the experience needed for this developmental process. Animals were then repeatedly exposed to different configurations of visual and auditory stimuli (e.g. spatiotemporally congruent or spatially disparate) that varied on each side of space and their behavior was assessed using a detection/localization task. Animals showed enhanced performance to stimuli consistent with the experience provided: congruent stimuli elicited enhanced behaviors where spatially congruent cross-modal experience was provided, and spatially disparate stimuli elicited enhanced behaviors where spatially disparate cross-modal experience was provided. Cross-modal configurations not consistent with experience did not enhance responses. The presumptive benefit of such flexibility in the multisensory developmental process is to sensitize neural circuits (and the behaviors they control) to the features of the environment in which they will function. These experiments reveal that these processes have a high degree of flexibility, such that two (conflicting) multisensory principles can be implemented by cross-modal experience on opposite sides of space even within the same animal.

The emergence of the multisensory brain: From the womb to the first steps

The becoming of the human being is a multisensory process that starts in the womb. By integrating spontaneous neuronal activity with inputs from the external world, the developing brain learns to make sense of itself through multiple sensory experiences. Over the past ten years, advances in neuroimaging and electrophysiological techniques have allowed the exploration of the neural correlates of multisensory processing in the newborn and infant brain, thus adding an important piece of information to behavioral evidence of early sensitivity to multisensory events. Here, we review recent behavioral and neuroimaging findings to document the origins and early development of multisensory processing, particularly showing that the human brain appears naturally tuned to multisensory events at birth, which requires multisensory experience to fully mature. We conclude the review by highlighting the potential uses and benefits of multisensory interventions in promoting healthy development by discussing emerging studies in preterm infants.

Multisensory Calibration: A Variety of Slow and Fast Brain Processes Throughout the Lifespan

From before we are born, throughout development, adulthood, and aging, we are immersed in a multisensory world. At each of these stages, our sensory cues are constantly changing, due to body, brain, and environmental changes. While integration of information from our different sensory cues improves precision, this only improves accuracy if the underlying cues are unbiased. Thus, multisensory calibration is a vital and ongoing process. To meet this grand challenge, our brains have evolved a variety of mechanisms. First, in response to a systematic discrepancy between sensory cues (without external feedback) the cues calibrate one another (unsupervised calibration). Second, multisensory function is calibrated to external feedback (supervised calibration). These two mechanisms superimpose. While the former likely reflects a lower level mechanism, the latter likely reflects a higher level cognitive mechanism. Indeed, neural correlates of supervised multisensory calibration in monkeys were found in higher level multisensory cortical area VIP, but not in the relatively lower level multisensory area MSTd. In addition, even without a cue discrepancy (e.g., when experiencing stimuli from different sensory cues in series) the brain monitors supra-modal statistics of events in the environment and adapts perception cross-modally. This too comprises a variety of mechanisms, including confirmation bias to prior choices, and lower level cross-sensory adaptation. Further research into the neuronal underpinnings of the broad and diverse functions of multisensory calibration, with improved synthesis of theories is needed to attain a more comprehensive understanding of multisensory brain function.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Multisensory decisions from self to world

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Higher synchronization stability with piano experience: relationship with finger and presentation modality

BACKGROUND

Synchronous finger tapping to external sensory stimuli is more stable for audiovisual combined stimuli than sole auditory or visual stimuli. In addition, piano players are superior in synchronous tapping and manipulating the ring and little fingers as compared to inexperienced individuals. However, it is currently unknown whether the ability to synchronize to external sensory stimuli with the ring finger is at the level of the index finger in piano players. The aim of this study was to compare the effect of piano experience on synchronization stability between the index and ring fingers using auditory, visual, and audiovisual combined stimuli.

METHODS

Thirteen piano players and thirteen novices participated in this study. They were instructed to tap with their index or ring finger synchronously to auditory, visual, and audiovisual combined stimuli. The stimuli were presented from an electronic metronome at 1 Hz, and the tapping was performed 30 times in each condition. We analyzed standard deviation of intervals between the stimulus onset and the tap onset as synchronization stability.

RESULTS

Synchronization stability for visual stimuli was lower during ring than index finger tapping in novices; however, this decline was absent in piano players. Also, piano players showed the higher synchronization stability for audiovisual combined stimuli than sole visual and auditory stimuli when tapping with the index finger. On the other hand, in novices, synchronization stability was higher for audiovisual combined stimuli than only visual stimuli.

CONCLUSIONS

These findings suggest that improvements of both sensorimotor processing and finger motor control by piano practice would contribute to superior synchronization stability.

Noise-rearing precludes the behavioral benefits of multisensory integration

Concordant visual-auditory stimuli enhance the responses of individual superior colliculus (SC) neurons. This neuronal capacity for "multisensory integration" is not innate: it is acquired only after substantial cross-modal (e.g. auditory-visual) experience. Masking transient auditory cues by raising animals in omnidirectional sound ("noise-rearing") precludes their ability to obtain this experience and the ability of the SC to construct a normal multisensory (auditory-visual) transform. SC responses to combinations of concordant visual-auditory stimuli are depressed, rather than enhanced. The present experiments examined the behavioral consequence of this rearing condition in a simple detection/localization task. In the first experiment, the auditory component of the concordant cross-modal pair was novel, and only the visual stimulus was a target. In the second experiment, both component stimuli were targets. Noise-reared animals failed to show multisensory performance benefits in either experiment. These results reveal a close parallel between behavior and single neuron physiology in the multisensory deficits that are induced when noise disrupts early visual-auditory experience.

Resolution of impaired multisensory processing in autism and the cost of switching sensory modality

Children with autism spectrum disorders (ASD) exhibit alterations in multisensory processing, which may contribute to the prevalence of social and communicative deficits in this population. Resolution of multisensory deficits has been observed in teenagers with ASD for complex, social speech stimuli; however, whether this resolution extends to more basic multisensory processing deficits remains unclear. Here, in a cohort of 364 participants we show using simple, non-social audiovisual stimuli that deficits in multisensory processing observed in high-functioning children and teenagers with ASD are not evident in adults with the disorder. Computational modelling indicated that multisensory processing transitions from a default state of competition to one of facilitation, and that this transition is delayed in ASD. Further analysis revealed group differences in how sensory channels are weighted, and how this is impacted by preceding cross-sensory inputs. Our findings indicate that there is a complex and dynamic interplay among the sensory systems that differs considerably in individuals with ASD.

Crosse et al. study a cohort of 364 participants with autism spectrum disorders (ASD) and matched controls, and show that deficits in multisensory processing observed in high-functioning children and teenagers with ASD are not evident in adults with the disorder. Using computational modelling they go on to demonstrate that there is a delayed transition of multisensory processing from a default state of competition to one of facilitation in ASD, as well as differences in sensory weighting and the ability to switch between sensory modalities, which sheds light on the interplay among sensory systems that differ in ASD individuals.

On the relationship between maps and domains in inferotemporal cortex

How does the brain encode information about the environment? Decades of research have led to the pervasive notion that the object-processing pathway in primate cortex consists of multiple areas that are each specialized to process different object categories (such as faces, bodies, hands, non-face objects and scenes). The anatomical consistency and modularity of these regions have been interpreted as evidence that these regions are innately specialized. Here, we propose that ventral-stream modules do not represent clusters of circuits that each evolved to process some specific object category particularly important for survival, but instead reflect the effects of experience on a domain-general architecture that evolved to be able to adapt, within a lifetime, to its particular environment. Furthermore, we propose that the mechanisms underlying the development of domains are both evolutionarily old and universal across cortex. Topographic maps are fundamental, governing the development of specializations across systems, providing a framework for brain organization.

Neuroplasticity and Crossmodal Connectivity in the Normal, Healthy Brain

Objective

Neuroplasticity enables the brain to establish new crossmodal connections or reorganize old connections which are essential to perceiving a multisensorial world. The intent of this review is to identify and summarize the current developments in neuroplasticity and crossmodal connectivity, and deepen understanding of how crossmodal connectivity develops in the normal, healthy brain, highlighting novel perspectives about the principles that guide this connectivity.

Methods

To the above end, a narrative review is carried out. The data documented in prior relevant studies in neuroscience, psychology and other related fields available in a wide range of prominent electronic databases are critically assessed, synthesized, interpreted with qualitative rather than quantitative elements, and linked together to form new propositions and hypotheses about neuroplasticity and crossmodal connectivity.

Results

Three major themes are identified. First, it appears that neuroplasticity operates by following eight fundamental principles and crossmodal integration operates by following three principles. Second, two different forms of crossmodal connectivity, namely direct crossmodal connectivity and indirect crossmodal connectivity, are suggested to operate in both unisensory and multisensory perception. Third, three principles possibly guide the development of crossmodal connectivity into adulthood. These are labeled as the principle of innate crossmodality, the principle of evolution-driven 'neuromodular' reorganization and the principle of multimodal experience. These principles are combined to develop a three-factor interaction model of crossmodal connectivity.

Conclusions

The hypothesized principles and the proposed model together advance understanding of neuroplasticity, the nature of crossmodal connectivity, and how such connectivity develops in the normal, healthy brain.

Multisensory enhancement of overt behavior requires multisensory experience

The superior colliculus (SC) is richly endowed with neurons that integrate cues from different senses to enhance their physiological responses and the overt behaviors they mediate. However, in the absence of experience with cross-modal combinations (e.g., visual-auditory), they fail to develop this characteristic multisensory capability: Their multisensory responses are no greater than their most effective unisensory responses. Presumably, this impairment in neural development would be reflected as corresponding impairments in SC-mediated behavioral capabilities such as detection and localization performance. Here, we tested that assumption directly in cats raised to adulthood in darkness. They, along with a normally reared cohort, were trained to approach brief visual or auditory stimuli. The animals were then tested with these stimuli individually and in combination under ambient light conditions consistent with their rearing conditions and home environment as well as under the opposite lighting condition. As expected, normally reared animals detected and localized the cross-modal combinations significantly better than their individual component stimuli. However, dark-reared animals showed significant defects in multisensory detection and localization performance. The results indicate that a physiological impairment in single multisensory SC neurons is predictive of an impairment in overt multisensory behaviors.

The postnatal development of MT, V1, LGN, pulvinar and SC in prosimian galagos (Otolemur garnettii)

Considerable evidence supports the premise that the visual system of primates develops hierarchically, with primary visual cortex developing structurally and functionally first, thereby influencing the subsequent development of higher cortical areas. An apparent exception is the higher order middle temporal visual area (MT), which appears to be histologically distinct near the time of birth in marmosets. Here we used a number of histological and immunohistological markers to evaluate the maturation of cortical and subcortical components of the visual system in galagos ranging from newborns to adults. Galagos are representative of the large strepsirrhine branch of primate evolution, and studies of these primates help identify brain features that are broadly similar across primate taxa. The histological results support the view that MT is functional at or near the time of birth, as is primary visual cortex. Likewise, the superior colliculus, dorsal lateral geniculate nucleus, and the posterior nucleus of the pulvinar are well-developed by birth. Thus, these subcortical structures likely provide visual information directly or indirectly to cortex in newborn galagos. We conclude that MT resembles a primary sensory area by developing early, and that the early development of MT may influence the subsequent development of dorsal stream visual areas.

Visuo-haptic transfer for object recognition in children with peripheral visual impairment

It is well known how early visual experience is critical for the development of multisensory processing abilities, and for this reason an early vision impairment could hinder the transfer of different sensory information during the exploration and recognition of the surrounding environment. Recently, we verified that visuo-haptic transfer for object recognition emerges early in typically developing children but matures slowly during the school-age period. Subsequently we verified the presence of a slower trend of development in unisensory and multisensory skills in children with early abnormal motor and sensory experiences due to brain lesions. Now, we investigated unimodal visual information, unimodal haptic information and visuo-haptic information transfer in children with a diagnosis of low-vision, due to congenital visual impairment. Unimodal and bimodal processes for object recognition were explored in 11 children with low-vision and the results were matched with those of 22 controls. Participants were tested using a clinical protocol involving visual exploration of black-and-white photographs of common objects, haptic exploration of real objects and visuo-haptic transfer of these two types of information. Results show a normal development in haptic unisensory processing in children with low vision and a significant difference in multisensory transfer between the two groups. In children with visual impairment, multisensory processes do not facilitate the recognition of common objects as in typical children, probably because early visual impairment may impact the cross-sensory calibration of vision and touch.

Approaches to Understanding Multisensory Dysfunction in Autism Spectrum Disorder

Abnormal sensory responses are a DSM-5 symptom of autism spectrum disorder (ASD), and research findings demonstrate altered sensory processing in ASD. Beyond difficulties with processing information within single sensory domains, including both hypersensitivity and hyposensitivity, difficulties in multisensory processing are becoming a core issue of focus in ASD. These difficulties may be targeted by treatment approaches such as "sensory integration," which is frequently applied in autism treatment but not yet based on clear evidence. Recently, psychophysical data have emerged to demonstrate multisensory deficits in some children with ASD. Unlike deficits in social communication, which are best understood in humans, sensory and multisensory changes offer a tractable marker of circuit dysfunction that is more easily translated into animal model systems to probe the underlying neurobiological mechanisms. Paralleling experimental paradigms that were previously applied in humans and larger mammals, we and others have demonstrated that multisensory function can also be examined behaviorally in rodents. Here, we review the sensory and multisensory difficulties commonly found in ASD, examining laboratory findings that relate these findings across species. Next, we discuss the known neurobiology of multisensory integration, drawing largely on experimental work in larger mammals, and extensions of these paradigms into rodents. Finally, we describe emerging investigations into multisensory processing in genetic mouse models related to autism risk. By detailing findings from humans to mice, we highlight the advantage of multisensory paradigms that can be easily translated across species, as well as the potential for rodent experimental systems to reveal opportunities for novel treatments. LAY SUMMARY: Sensory and multisensory deficits are commonly found in ASD and may result in cascading effects that impact social communication. By using similar experiments to those in humans, we discuss how studies in animal models may allow an understanding of the brain mechanisms that underlie difficulties in multisensory integration, with the ultimate goal of developing new treatments. Autism Res 2020, 13: 1430-1449. © 2020 International Society for Autism Research, Wiley Periodicals, Inc.

Using superior colliculus principles of multisensory integration to reverse hemianopia

The diversity of our senses conveys many advantages; it enables them to compensate for one another when needed, and the information they provide about a common event can be integrated to facilitate its processing and, ultimately, adaptive responses. These cooperative interactions are produced by multisensory neurons. A well-studied model in this context is the multisensory neuron in the output layers of the superior colliculus (SC). These neurons integrate and amplify their cross-modal (e.g., visual-auditory) inputs, thereby enhancing the physiological salience of the initiating event and the probability that it will elicit SC-mediated detection, localization, and orientation behavior. Repeated experience with the same visual-auditory stimulus can also increase the neuron's sensitivity to these individual inputs. This observation raised the possibility that such plasticity could be engaged to restore visual responsiveness when compromised. For example, unilateral lesions of visual cortex compromise the visual responsiveness of neurons in the multisensory output layers of the ipsilesional SC and produces profound contralesional blindness (hemianopia). The possibility that multisensory plasticity could restore the visual responses of these neurons, and reverse blindness, was tested in the cat model of hemianopia. Hemianopic subjects were repeatedly presented with spatiotemporally congruent visual-auditory stimulus pairs in the blinded hemifield on a daily or weekly basis. After several weeks of this multisensory exposure paradigm, visual responsiveness was restored in SC neurons and behavioral responses were elicited by visual stimuli in the previously blind hemifield. The constraints on the effectiveness of this procedure proved to be the same as those constraining SC multisensory plasticity: whereas repetitions of a congruent visual-auditory stimulus was highly effective, neither exposure to its individual component stimuli, nor to these stimuli in non-congruent configurations was effective. The restored visual responsiveness proved to be robust, highly competitive with that in the intact hemifield, and sufficient to support visual discrimination.

Multisensory Integration as a Window into Orderly and Disrupted Cognition and Communication

During our everyday lives, we are confronted with a vast amount of information from several sensory modalities. This multisensory information needs to be appropriately integrated for us to effectively engage with and learn from our world. Research carried out over the last half century has provided new insights into the way such multisensory processing improves human performance and perception; the neurophysiological foundations of multisensory function; the time course for its development; how multisensory abilities differ in clinical populations; and, most recently, the links between multisensory processing and cognitive abilities. This review summarizes the extant literature on multisensory function in typical and atypical circumstances, discusses the implications of the work carried out to date for theory and research, and points toward next steps for advancing the field.

Active touch facilitates object size perception in children but not adults: A multisensory event related potential study

In order to increase perceptual precision the adult brain dynamically combines redundant information from different senses depending on their reliability. During object size estimation, for example, visual, auditory and haptic information can be integrated to increase the precision of the final size estimate. Young children, however, do not integrate sensory information optimally and instead rely on active touch. Whether this early haptic dominance is reflected in age-related differences in neural mechanisms and whether it is driven by changes in bottom-up perceptual or top-down attentional processes has not yet been investigated. Here, we recorded event-related-potentials from a group of adults and children aged 5-7 years during an object size perception task using auditory, visual and haptic information. Multisensory information was presented either congruently (conveying the same information) or incongruently (conflicting information). No behavioral responses were required from participants. When haptic size information was available via actively tapping the objects, response amplitudes in the mid-parietal area were significantly reduced by information congruency in children but not in adults between 190 ms-250 ms and 310 ms-370 ms. These findings indicate that during object size perception only children's brain activity is modulated by active touch supporting a neural maturational shift from sensory dominance in early childhood to optimal multisensory benefit in adulthood.

Evidence for Brainstem Contributions to Autism Spectrum Disorders

Autism spectrum disorder (ASD) is a neurodevelopmental condition that affects one in 59 children in the United States. Although there is a mounting body of knowledge of cortical and cerebellar contributions to ASD, our knowledge about the early developing brainstem in ASD is only beginning to accumulate. Understanding how brainstem neurotransmission is implicated in ASD is important because many of this condition’s sensory and motor symptoms are consistent with brainstem pathology. Therefore, the purpose of this review was to integrate epidemiological, behavioral, histological, neuroimaging, and animal evidence of brainstem contributions to ASD. Because ASD is a neurodevelopmental condition, we examined the available data through a lens of hierarchical brain development. The review of the literature suggests that developmental alterations of the brainstem could have potential cascading effects on cortical and cerebellar formation, ultimately leading to ASD symptoms. This view is supported by human epidemiology findings and data from animal models of ASD, showing that perturbed development of the brainstem substructures, particularly during the peak formation of the brainstem’s monoaminergic centers, may relate to ASD or ASD-like behaviors. Furthermore, we review evidence from human histology, psychophysiology, and neuroimaging suggesting that brainstem development and maturation may be atypical in ASD and may be related to key ASD symptoms, such as atypical sensorimotor features and social responsiveness. From this review there emerges the need of future research to validate early detection of the brainstem-based somatosensory and psychophysiological behaviors that emerge in infancy, and to examine the brainstem across the life span, while accounting for age. In all, there is preliminary evidence for brainstem involvement in ASD, but a better understanding of the brainstem’s role would likely pave the way for earlier diagnosis and treatment of ASD.

Temporal Cues Influence Space Estimations in Visually Impaired Individuals

Many works have highlighted enhanced auditory processing in blind individuals, suggesting that they compensate for lack of vision with greater sensitivity of the other senses. Few years ago, we demonstrated severely impaired auditory precision in congenitally blind individuals performing an auditory spatial metric task: their thresholds for bisecting three consecutive spatially distributed sounds were seriously compromised, ranging from three times typical thresholds to total randomness. Here, we show that the deficit disappears if blind individuals are presented with coherent temporal and spatial cues. More interestingly, when the audio information is presented in conflict for space and time, sighted individuals are unaffected by the perturbation, whereas blind individuals are strongly attracted by the temporal cue. These results highlight that temporal cues influence space estimations in blind participants, suggesting for the first time that blind individuals use temporal information to infer spatial environmental coordinates.

•

Blind individuals are not able to perform auditory spatial metric tasks

•

Their deficit disappears when coherent temporal and spatial cues are presented

•

In some cases, blind people use temporal cues to infer spatial coordinates

The redeployment of attention to the mouth of a talking face during the second year of life

Previous studies have found that when monolingual infants are exposed to a talking face speaking in a native language, 8- and 10-month-olds attend more to the talker's mouth, whereas 12-month-olds no longer do so. It has been hypothesized that the attentional focus on the talker's mouth at 8 and 10 months of age reflects reliance on the highly salient audiovisual (AV) speech cues for the acquisition of basic speech forms and that the subsequent decline of attention to the mouth by 12 months of age reflects the emergence of basic native speech expertise. Here, we investigated whether infants may redeploy their attention to the mouth once they fully enter the word-learning phase. To test this possibility, we recorded eye gaze in monolingual English-learning 14- and 18-month-olds while they saw and heard a talker producing an English or Spanish utterance in either an infant-directed (ID) or adult-directed (AD) manner. Results indicated that the 14-month-olds attended more to the talker's mouth than to the eyes when exposed to the ID utterance and that the 18-month-olds attended more to the talker's mouth when exposed to the ID and the AD utterance. These results show that infants redeploy their attention to a talker's mouth when they enter the word acquisition phase and suggest that infants rely on the greater perceptual salience of redundant AV speech cues to acquire their lexicon.

From multisensory integration in peripersonal space to bodily self-consciousness: from statistical regularities to statistical inference

Integrating information across sensory systems is a critical step toward building a cohesive representation of the environment and one's body, and as illustrated by numerous illusions, scaffolds subjective experience of the world and self. In the last years, classic principles of multisensory integration elucidated in the subcortex have been translated into the language of statistical inference understood by the neocortical mantle. Most importantly, a mechanistic systems-level description of multisensory computations via probabilistic population coding and divisive normalization is actively being put forward. In parallel, by describing and understanding bodily illusions, researchers have suggested multisensory integration of bodily inputs within the peripersonal space as a key mechanism in bodily self-consciousness. Importantly, certain aspects of bodily self-consciousness, although still very much a minority, have been recently casted under the light of modern computational understandings of multisensory integration. In doing so, we argue, the field of bodily self-consciousness may borrow mechanistic descriptions regarding the neural implementation of inference computations outlined by the multisensory field. This computational approach, leveraged on the understanding of multisensory processes generally, promises to advance scientific comprehension regarding one of the most mysterious questions puzzling humankind, that is, how our brain creates the experience of a self in interaction with the environment.

Multisensory cortical processing and dysfunction across the neuropsychiatric spectrum

Sensory processing is affected in multiple neuropsychiatric disorders like schizophrenia and autism spectrum disorders. Genetic and environmental factors guide the formation and fine-tuning of brain circuitry necessary to receive, organize, and respond to sensory input in order to behave in a meaningful and consistent manner. During certain developmental stages the brain is sensitive to intrinsic and external factors. For example, disturbed expression levels of certain risk genes during critical neurodevelopmental periods may lead to exaggerated brain plasticity processes within the sensory circuits, and sensory stimulation immediately after birth contributes to fine-tuning of these circuits. Here, the neurodevelopmental trajectory of sensory circuit development will be described and related to some example risk gene mutations that are found in neuropsychiatric disorders. Subsequently, the flow of sensory information through these circuits and the relationship to synaptic plasticity will be described. Research focusing on the combined analyses of neural circuit development and functioning are necessary to expand our understanding of sensory processing and behavioral deficits that are relevant across the neuropsychiatric spectrum.

Ripe for solution: Delayed development of multisensory processing in autism and its remediation

Difficulty integrating inputs from different sensory sources is commonly reported in individuals with Autism Spectrum Disorder (ASD). Accumulating evidence consistently points to altered patterns of behavioral reactions and neural activity when individuals with ASD observe or act upon information arriving through multiple sensory systems. For example, impairments in the integration of seen and heard speech appear to be particularly acute, with obvious implications for interpersonal communication. Here, we explore the literature on multisensory processing in autism with a focus on developmental trajectories. While much remains to be understood, some consistent observations emerge. Broadly, sensory integration deficits are found in children with an ASD whereas these appear to be much ameliorated, or even fully recovered, in older teenagers and adults on the spectrum. This protracted delay in the development of multisensory processing raises the possibility of applying early intervention strategies focused on multisensory integration, to accelerate resolution of these functions. We also consider how dysfunctional cross-sensory oscillatory neural communication may be one key pathway to impaired multisensory processing in ASD.

Multisensory-Based Rehabilitation Approach: Translational Insights from Animal Models to Early Intervention

Multisensory processes permit combinations of several inputs, coming from different sensory systems, allowing for a coherent representation of biological events and facilitating adaptation to environment. For these reasons, their application in neurological and neuropsychological rehabilitation has been enhanced in the last decades. Recent studies on animals and human models have indicated that, on one hand multisensory integration matures gradually during post-natal life and development is closely linked to environment and experience and, on the other hand, that modality-specific information seems to do not benefit by redundancy across multiple sense modalities and is more readily perceived in unimodal than in multimodal stimulation. In this review, multisensory process development is analyzed, highlighting clinical effects in animal and human models of its manipulation for rehabilitation of sensory disorders. In addition, new methods of early intervention based on multisensory-based rehabilitation approach and their applications on different infant populations at risk of neurodevelopmental disabilities are discussed.

Audiovisual integration in near and far space: effects of changes in distance and stimulus effectiveness

A factor that is often not considered in multisensory research is the distance from which information is presented. Interestingly, various studies have shown that the distance at which information is presented can modulate the strength of multisensory interactions. In addition, our everyday multisensory experience in near and far space is rather asymmetrical in terms of retinal image size and stimulus intensity. This asymmetry is the result of the relation between the stimulus-observer distance and its retinal image size and intensity: an object that is further away is generally smaller on the retina as compared to the same object when it is presented nearer. Similarly, auditory intensity decreases as the distance from the observer increases. We investigated how each of these factors alone, and their combination, affected audiovisual integration. Unimodal and bimodal stimuli were presented in near and far space, with and without controlling for distance-dependent changes in retinal image size and intensity. Audiovisual integration was enhanced for stimuli that were presented in far space as compared to near space, but only when the stimuli were not corrected for visual angle and intensity. The same decrease in intensity and retinal size in near space did not enhance audiovisual integration, indicating that these results cannot be explained by changes in stimulus efficacy or an increase in distance alone, but rather by an interaction between these factors. The results are discussed in the context of multisensory experience and spatial uncertainty, and underline the importance of studying multisensory integration in the depth space.

Multisensory body representation in autoimmune diseases

Body representation has been linked to the processing and integration of multisensory signals. An outstanding example of the pivotal role played by multisensory mechanisms in body representation is the Rubber Hand Illusion (RHI). In this paradigm, multisensory stimulation induces a sense of ownership over a fake limb. Previous work has shown high interindividual differences in the susceptibility to the RHI. The origin of this variability remains largely unknown. Given the tight and bidirectional communication between the brain and the immune system, we predicted that the origin of this variability could be traced, in part, to the immune system’s functioning, which is altered by several clinical conditions, including Coeliac Disease (CD). Consistent with this prediction, we found that the Rubber Hand Illusion is stronger in CD patients as compared to healthy controls. We propose a biochemical mechanism accounting for the dependency of multisensory body representation upon the Immune system. Our finding has direct implications for a range of neurological, psychiatric and immunological conditions where alterations of multisensory integration, body representation and dysfunction of the immune system co-exist.

Predictive coding and multisensory integration: an attentional account of the multisensory mind

Multisensory integration involves a host of different cognitive processes, occurring at different stages of sensory processing. Here I argue that, despite recent insights suggesting that multisensory interactions can occur at very early latencies, the actual integration of individual sensory traces into an internally consistent mental representation is dependent on both top–down and bottom–up processes. Moreover, I argue that this integration is not limited to just sensory inputs, but that internal cognitive processes also shape the resulting mental representation. Studies showing that memory recall is affected by the initial multisensory context in which the stimuli were presented will be discussed, as well as several studies showing that mental imagery can affect multisensory illusions. This empirical evidence will be discussed from a predictive coding perspective, in which a central top–down attentional process is proposed to play a central role in coordinating the integration of all these inputs into a coherent mental representation.

Body perception, awareness, and illusions

Perceiving a body is a phenomenal experience completely different from experiencing a body as one's own body. Visual presentation of bodies or body parts recruits several occipitotemporal regions in the brain. Are these activations sufficient in order to change the phenomenal status of a body in one's own body? In this paper, I will review consolidated experimental evidence showing that the feeling of owning a body is not limited to the vision of a body, rather it is the result of a complex interaction between interoception, exteroception, and pre-existing body templates. To illustrate this complex interplay, I will take advantage of the so-called bodily illusions, referring to controlled illusory generation of unusual bodily feeling. These feelings include having a supernumerary limb, or lacking an arm, or feeling like you do not really have a body, or feeling that you do not really control a certain part of your body, or that your body is not really yours. In the last 15 years more than 150 empirical studies on body illusions have been published (

SOURCE

Pubmed, June 2014). These studies, using different technologies, are largely responsible for contributed our current understanding of bodily self-consciousness. WIREs Cogn Sci 2014, 5:551-560. doi: 10.1002/wcs.1309 For further resources related to this article, please visit the WIREs website.

CONFLICT OF INTEREST

The author has declared no conflicts of interest for this article.

Crossmodal integration: a glimpse into the development of sensory remapping

Correctly localising sensory stimuli in space is a formidable challenge for the newborn brain. A new study provides a first glimpse into how human brain mechanisms for sensory remapping develop in the first year of life.

When vision is not an option: children's integration of auditory and haptic information is suboptimal

When visual information is available, human adults, but not children, have been shown to reduce sensory uncertainty by taking a weighted average of sensory cues. In the absence of reliable visual information (e.g. extremely dark environment, visual disorders), the use of other information is vital. Here we ask how humans combine haptic and auditory information from childhood. In the first experiment, adults and children aged 5 to 11 years judged the relative sizes of two objects in auditory, haptic, and non-conflicting bimodal conditions. In Experiment 2, different groups of adults and children were tested in non-conflicting and conflicting bimodal conditions. In Experiment 1, adults reduced sensory uncertainty by integrating the cues optimally, while children did not. In Experiment 2, adults and children used similar weighting strategies to solve audio–haptic conflict. These results suggest that, in the absence of visual information, optimal integration of cues for discrimination of object size develops late in childhood.

Early experience and multisensory perceptual narrowing

Perceptual narrowing reflects the effects of early experience and contributes in key ways to perceptual and cognitive development. Previous studies have found that unisensory perceptual sensitivity in young infants is broadly tuned such that they can discriminate native as well as non-native sensory inputs but that it is more narrowly tuned in older infants such that they only respond to native inputs. Recently, my coworkers and I discovered that multisensory perceptual sensitivity narrows as well. The present article reviews this new evidence in the general context of multisensory perceptual development and the effects of early experience. Together, the evidence on unisensory and multisensory narrowing shows that early experience shapes the emergence of perceptual specialization and expertise.

Inferences about infants’ visual brain mechanisms

We discuss hypotheses that link the measurements we can make with infants to inferences about their developing neural mechanisms. First, we examine evidence from the sensitivity to visual stimulus properties seen in infants’ responses, using both electrophysiological measures (transient and steady-state recordings of visual evoked potentials/visual event-related potentials) and behavioral measures and compare this with the sensitivity of brain processes, known from data on mammalian neurophysiology and human neuroimaging. The evidence for multiple behavioral systems with different patterns of visual sensitivity is discussed. Second, we consider the analogies which can be made between infants’ behavior and that of adults with identified brain damage, and extend these links to hypothesize about the brain basis of visual deficits in infants and children with developmental disorders. Last, we consider how these lines of data might allow us to form “inverse linking hypotheses” about infants’ visual experience.

Modeling the minimal newborn's intersubjective mind: the visuotopic-somatotopic alignment hypothesis in the superior colliculus

The question whether newborns possess inborn social skills is a long debate in developmental psychology. Fetal behavioral and anatomical observations show evidences for the control of eye movements and facial behaviors during the third trimester of pregnancy whereas specific sub-cortical areas, like the superior colliculus (SC) and the striatum appear to be functionally mature to support these behaviors. These observations suggest that the newborn is potentially mature for developing minimal social skills. In this manuscript, we propose that the mechanism of sensory alignment observed in SC is particularly important for enabling the social skills observed at birth such as facial preference and facial mimicry. In a computational simulation of the maturing superior colliculus connected to a simulated facial tissue of a fetus, we model how the incoming tactile information is used to direct visual attention toward faces. We suggest that the unisensory superficial visual layer (eye-centered) and the deep somatopic layer (face-centered) in SC are combined into an intermediate layer for visuo-tactile integration and that multimodal alignment in this third layer allows newborns to have a sensitivity to configuration of eyes and mouth. We show that the visual and tactile maps align through a Hebbian learning stage and and strengthen their synaptic links from each other into the intermediate layer. It results that the global network produces some emergent properties such as sensitivity toward the spatial configuration of face-like patterns and the detection of eyes and mouth movement.

Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons

The richness of perceptual experience, as well as its usefulness for guiding behaviour, depends on the synthesis of information across multiple senses. Recent decades have witnessed a surge in our understanding of how the brain combines sensory cues. Much of this research has been guided by one of two distinct approaches: one is driven primarily by neurophysiological observations, and the other is guided by principles of mathematical psychology and psychophysics. Conflicting results and interpretations have contributed to a conceptual gap between psychophysical and physiological accounts of cue integration, but recent studies of visual-vestibular cue integration have narrowed this gap considerably.

Development of cortical influences on superior colliculus multisensory neurons: effects of dark-rearing

Rearing cats from birth to adulthood in darkness prevents neurons in the superior colliculus (SC) from developing the capability to integrate visual and non-visual (e.g. visual-auditory) inputs. Presumably, this developmental anomaly is due to a lack of experience with the combination of those cues, which is essential to form associative links between them. The visual-auditory multisensory integration capacity of SC neurons has also been shown to depend on the functional integrity of converging visual and auditory inputs from the ipsilateral association cortex. Disrupting these cortico-collicular projections at any stage of life results in a pattern of outcomes similar to those found after dark-rearing; SC neurons respond to stimuli in both sensory modalities, but cannot integrate the information they provide. Thus, it is possible that dark-rearing compromises the development of these descending tecto-petal connections and the essential influences they convey. However, the results of the present experiments, using cortical deactivation to assess the presence of cortico-collicular influences, demonstrate that dark-rearing does not prevent the association cortex from developing robust influences over SC multisensory responses. In fact, dark-rearing may increase their potency over that observed in normally-reared animals. Nevertheless, their influences are still insufficient to support SC multisensory integration. It appears that cross-modal experience shapes the cortical influence to selectively enhance responses to cross-modal stimulus combinations that are likely to be derived from the same event. In the absence of this experience, the cortex develops an indiscriminate excitatory influence over its multisensory SC target neurons.

Hebbian mechanisms help explain development of multisensory integration in the superior colliculus: a neural network model

The superior colliculus (SC) integrates relevant sensory information (visual, auditory, somatosensory) from several cortical and subcortical structures, to program orientation responses to external events. However, this capacity is not present at birth, and it is acquired only through interactions with cross-modal events during maturation. Mathematical models provide a quantitative framework, valuable in helping to clarify the specific neural mechanisms underlying the maturation of the multisensory integration in the SC. We extended a neural network model of the adult SC (Cuppini et al., Front Integr Neurosci 4:1-15, 2010) to describe the development of this phenomenon starting from an immature state, based on known or suspected anatomy and physiology, in which: (1) AES afferents are present but weak, (2) Responses are driven from non-AES afferents, and (3) The visual inputs have a marginal spatial tuning. Sensory experience was modeled by repeatedly presenting modality-specific and cross-modal stimuli. Synapses in the network were modified by simple Hebbian learning rules. As a consequence of this exposure, (1) Receptive fields shrink and come into spatial register, and (2) SC neurons gained the adult characteristic integrative properties: enhancement, depression, and inverse effectiveness. Importantly, the unique architecture of the model guided the development so that integration became dependent on the relationship between the cortical input and the SC. Manipulations of the statistics of the experience during the development changed the integrative profiles of the neurons, and results matched well with the results of physiological studies.

Neural correlates of perceptual narrowing in cross-species face-voice matching

Integrating the multisensory features of talking faces is critical to learning and extracting coherent meaning from social signals. While we know much about the development of these capacities at the behavioral level, we know very little about the underlying neural processes. One prominent behavioral milestone of these capacities is the perceptual narrowing of face-voice matching, whereby young infants match faces and voices across species, but older infants do not. In the present study, we provide neurophysiological evidence for developmental decline in cross-species face-voice matching. We measured event-related brain potentials (ERPs) while 4- and 8-month-old infants watched and listened to congruent and incongruent audio-visual presentations of monkey vocalizations and humans mimicking monkey vocalizations. The ERP results indicated that younger infants distinguished between the congruent and the incongruent faces and voices regardless of species, whereas in older infants, the sensitivity to multisensory congruency was limited to the human face and voice. Furthermore, with development, visual and frontal brain processes and their functional connectivity became more sensitive to the congruence of human faces and voices relative to monkey faces and voices. Our data show the neural correlates of perceptual narrowing in face-voice matching and support the notion that postnatal experience with species identity is associated with neural changes in multisensory processing (Lewkowicz & Ghazanfar, 2009).

A rational analysis of the acquisition of multisensory representations

How do people learn multisensory, or amodal, representations, and what consequences do these representations have for perceptual performance? We address this question by performing a rational analysis of the problem of learning multisensory representations. This analysis makes use of a Bayesian nonparametric model that acquires latent multisensory features that optimally explain the unisensory features arising in individual sensory modalities. The model qualitatively accounts for several important aspects of multisensory perception: (a) it integrates information from multiple sensory sources in such a way that it leads to superior performances in, for example, categorization tasks; (b) its performances suggest that multisensory training leads to better learning than unisensory training, even when testing is conducted in unisensory conditions; (c) its multisensory representations are modality invariant; and (d) it predicts ''missing" sensory representations in modalities when the input to those modalities is absent. Our rational analysis indicates that all of these aspects emerge as part of the optimal solution to the problem of learning to represent complex multisensory environments.