The neural basis of multisensory integration in the midbrain: its organization and maturation

Peripersonal space: why so last-second?

A vast range of neurophysiological, neuropsychological and behavioural results in monkeys and humans have shown that the immediate surroundings of the body, also known as peripersonal space (PPS), are processed in a unique way. Three roles have been ascribed to PPS mechanisms: to react to threats, to avoid obstacles and to act on objects. However, in many circumstances, one does not wait for objects or agents to enter PPS to plan these behaviours. Typically, one has more chances to survive if one starts running away from the lion when one sees it in the distance than if it is a few steps away. PPS makes sense in shortsighted creatures but we are not such creatures. The crucial question is thus twofold: (i) why are these adaptive processes triggered only at the last second or even milliseconds? And (ii) what is their exact contribution, especially for defensive and navigational behaviours? Here, we propose that PPS mechanisms correspond to a plan B, useful in unpredictable situations or when other anticipatory mechanisms have failed. Furthermore, we argue that there are energetic, cognitive and behavioural costs to PPS mechanisms, which explain why this plan B is triggered only at the last second. This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Genetically defined neuron types underlying visuomotor transformation in the superior colliculus

The superior colliculus (SC) is a conserved midbrain structure that is important for transforming visual and other sensory information into motor actions. Decades of investigations in numerous species have made the SC and its nonmammalian homologue, the optic tectum, one of the best studied structures in the brain, with rich information now available regarding its anatomical organization, its extensive inputs and outputs and its important functions in many reflexive and cognitive behaviours. Excitingly, recent studies using modern genomic and physiological approaches have begun to reveal the diverse neuronal subtypes in the SC, as well as their unique functions in visuomotor transformation. Studies have also started to uncover how subtypes of SC neurons form intricate circuits to mediate visual processing and visually guided behaviours. Here, we review these recent discoveries on the cell types and neuronal circuits underlying visuomotor transformations mediated by the SC. We also highlight the important future directions made possible by these new developments.

Transthalamic Pathways for Cortical Function

The cerebral cortex contains multiple, distinct areas that individually perform specific computations. A particular strength of the cortex is the communication of signals between cortical areas that allows the outputs of these compartmentalized computations to influence and build on each other, thereby dramatically increasing the processing power of the cortex and its role in sensation, action, and cognition. Determining how the cortex communicates signals between individual areas is, therefore, critical for understanding cortical function. Historically, corticocortical communication was thought to occur exclusively by direct anatomical connections between areas that often sequentially linked cortical areas in a hierarchical fashion. More recently, anatomical, physiological, and behavioral evidence is accumulating indicating a role for the higher-order thalamus in corticocortical communication. Specifically, the transthalamic pathway involves projections from one area of the cortex to neurons in the higher-order thalamus that, in turn, project to another area of the cortex. Here, we consider the evidence for and implications of having two routes for corticocortical communication with an emphasis on unique processing available in the transthalamic pathway and the consequences of disorders and diseases that affect transthalamic communication.

Cross-modal enhancement of defensive behavior via parabigemino-collicular projections

Effective detection and avoidance from environmental threats are crucial for animals' survival. Integration of sensory cues associated with threats across different modalities can significantly enhance animals' detection and behavioral responses. However, the neural circuit-level mechanisms underlying the modulation of defensive behavior or fear response under simultaneous multimodal sensory inputs remain poorly understood. Here, we report in mice that bimodal looming stimuli combining coherent visual and auditory signals elicit more robust defensive/fear reactions than unimodal stimuli. These include intensified escape and prolonged hiding, suggesting a heightened defensive/fear state. These various responses depend on the activity of the superior colliculus (SC), while its downstream nucleus, the parabigeminal nucleus (PBG), predominantly influences the duration of hiding behavior. PBG temporally integrates visual and auditory signals and enhances the salience of threat signals by amplifying SC sensory responses through its feedback projection to the visual layer of the SC. Our results suggest an evolutionarily conserved pathway in defense circuits for multisensory integration and cross-modality enhancement.

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain

Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality. We record electroencephalogram (EEG) during a crossmodal audiovisual local-global oddball paradigm, in which the predictability of transitions between tones and images are manipulated at both the stimulus and sequence levels. To dissect the complex predictive signals in our EEG data, we employed a model-fitting approach to untangle neural interactions across modalities and hierarchies. The model-fitting result demonstrates that audiovisual integration occurs at both the levels of individual stimulus interactions and multi-stimulus sequences. Furthermore, we identify the spatio-spectro-temporal signatures of prediction-error signals across hierarchies and modalities, and reveal that auditory and visual prediction errors are rapidly redirected to the central-parietal electrodes during learning through alpha-band interactions. Our study suggests a crossmodal predictive coding mechanism where unimodal predictions are processed by distributed brain networks to form crossmodal knowledge.

Electrophysiological responses of audiovisual integration from infancy to adulthood

Our ability to merge information from different senses into a unified percept is a crucial perceptual process for efficient interaction with our multisensory environment. Yet, the developmental process underlying how the brain implements multisensory integration (MSI) remains poorly known. This cross-sectional study aims to characterize the developmental patterns of audiovisual events in 131 individuals aged from 3 months to 30 years. Electroencephalography (EEG) was recorded during a passive task, including simple auditory, visual, and audiovisual stimuli. In addition to examining age-related variations in MSI responses, we investigated Event-Related Potentials (ERPs) linked with auditory and visual stimulation alone. This was done to depict the typical developmental trajectory of unisensory processing from infancy to adulthood within our sample and to contextualize the maturation effects of MSI in relation to unisensory development. Comparing the neural response to audiovisual stimuli to the sum of the unisensory responses revealed signs of MSI in the ERPs, more specifically between the P2 and N2 components (P2 effect). Furthermore, adult-like MSI responses emerge relatively late in the development, around 8 years old. The automatic integration of simple audiovisual stimuli is a long developmental process that emerges during childhood and continues to mature during adolescence with ERP latencies decreasing with age.

Specific rules for time and space of multisensory plasticity in the superior colliculus

Cat superior colliculus (SC) neurons commonly combine information from different senses, which facilitates event detection and localization. Integration in SC multisensory neurons depends on the spatial and temporal relationships between cross-modal cues. Here, we revealed the parallel process of short-term plasticity in the temporal/spatial integration process during adulthood that adapts multisensory integration to reliable changes in environmental conditions. Short-term experience alters the temporal preferences of SC multisensory neurons, and this short-term plasticity in the temporal/spatial integration process is limited to changes in cross-modal timing (a factor commonly induced by events at different distances from the receiver). However, this plasticity was not evident in response to changes in the cross-modal spatial configuration.

The multisensory mind: a systematic review of multisensory integration processing in Anorexia and Bulimia Nervosa

Individuals with Anorexia Nervosa and Bulimia Nervosa present alterations in the way they experience their bodies. Body experience results from a multisensory integration process in which information from different sensory domains and spatial reference frames is combined into a coherent percept. Given the critical role of the body in the onset and maintenance of both Anorexia Nervosa and Bulimia Nervosa, we conducted a systematic review to examine multisensory integration abilities of individuals affected by these two conditions and investigate whether they exhibit impairments in crossmodal integration. We searched for studies evaluating crossmodal integration in individuals with a current diagnosis of Anorexia Nervosa and Bulimia Nervosa as compared to healthy individuals from both behavioral and neurobiological perspectives. A search of PubMed, PsycINFO, and Web of Sciences databases was performed to extract relevant articles. Of the 2348 studies retrieved, 911 were unique articles. After the screening, 13 articles were included. Studies revealed multisensory integration abnormalities in patients affected by Anorexia Nervosa; only one included individuals with Bulimia Nervosa and observed less severe impairments compared to healthy controls. Overall, results seemed to support the presence of multisensory deficits in Anorexia Nervosa, especially when integrating interoceptive and exteroceptive information. We proposed the Predictive Coding framework for understanding our findings and suggested future lines of investigation.

Eating, Smelling, and Seeing: Investigating Multisensory Integration and (In)congruent Stimuli while Eating in VR

Integrating taste in AR/VR applications has various promising use cases - from social eating to the treatment of disorders. Despite many successful AR/VR applications that alter the taste of beverages and food, the relationship between olfaction, gustation, and vision during the process of multisensory integration (MSI) has not been fully explored yet. Thus, we present the results of a study in which participants were confronted with congruent and incongruent visual and olfactory stimuli while eating a tasteless food product in VR. We were interested (1) if participants integrate bi-modal congruent stimuli and (2) if vision guides MSI during congruent/incongruent conditions. Our results contain three main findings: First, and surprisingly, participants were not always able to detect congruent visual-olfactory stimuli when eating a portion of tasteless food. Second, when confronted with tri-modal incongruent cues, a majority of participants did not rely on any of the presented cues when forced to identify what they eat; this includes vision which has previously been shown to dominate MSI. Third, although research has shown that basic taste qualities like sweetness, saltiness, or sourness can be influenced by congruent cues, doing so with more complex flavors (e.g., zucchini or carrot) proved to be harder to achieve. We discuss our results in the context of multimodal integration, and within the domain of multisensory AR/VR. Our results are a necessary building block for future human-food interaction in XR that relies on smell, taste, and vision and are foundational for applied applications such as affective AR/VR.

Multisensory perception is not influenced by previous concussion history in retired rugby union players

BACKGROUND

To assess whether concussion history adversely affects multisensory integration, we compared susceptibility to the Sound-Induced Flash Illusion (SIFI) in retired professional rugby players compared to controls.

METHODS

Retired professional rugby players ((N = 58) and retired international rowers (N = 26) completed a self-report concussion history questionnaire and the SIFI task. Susceptibility to the SIFI (i.e., perceiving two flashes in response to one flash paired with two beeps) was assessed at three stimulus onset asynchronies (70 ms, 150 ms or 230 ms).Logistic mixed-effects regression modeling was implemented to evaluate how athlete grouping, previous concussion history and total number of years playing sport, impacted the susceptibility to the SIFI task. The statistical significance of a fixed effect of interest was determined by a likelihood ratio test.

RESULTS

Former rugby players had significantly more self-reported concussions than the rower group (p < 0.001). There was no impact of athlete grouping (i.e., retired professional rugby players and retired international rowers), years participation in elite sport or concussion history on performance in the SIFI.

CONCLUSION

A career in professional rugby, concussion history or number of years participating in professional rugby was not found to be predictive of performance on the SIFI task.

Congruent aero-tactile stimuli bias perception of voicing continua

Multimodal integration is the formation of a coherent percept from different sensory inputs such as vision, audition, and somatosensation. Most research on multimodal integration in speech perception has focused on audio-visual integration. In recent years, audio-tactile integration has also been investigated, and it has been established that puffs of air applied to the skin and timed with listening tasks shift the perception of voicing by naive listeners. The current study has replicated and extended these findings by testing the effect of air puffs on gradations of voice onset time along a continuum rather than the voiced and voiceless endpoints of the original work. Three continua were tested: bilabial (“pa/ba”), velar (“ka/ga”), and a vowel continuum (“head/hid”) used as a control. The presence of air puffs was found to significantly increase the likelihood of choosing voiceless responses for the two VOT continua but had no effect on choices for the vowel continuum. Analysis of response times revealed that the presence of air puffs lengthened responses for intermediate (ambiguous) stimuli and shortened them for endpoint (non-ambiguous) stimuli. The slowest response times were observed for the intermediate steps for all three continua, but for the bilabial continuum this effect interacted with the presence of air puffs: responses were slower in the presence of air puffs, and faster in their absence. This suggests that during integration auditory and aero-tactile inputs are weighted differently by the perceptual system, with the latter exerting greater influence in those cases where the auditory cues for voicing are ambiguous.

Cross-modal facilitation of auditory discrimination in a frog

Stimulation in one sensory modality can affect perception in a separate modality, resulting in diverse effects including illusions in humans. This can also result in cross-modal facilitation, a process where sensory performance in one modality is improved by stimulation in another modality. For instance, a simple sound can improve performance in a visual task in both humans and cats. However, the range of contexts and underlying mechanisms that evoke such facilitation effects remain poorly understood. Here, we demonstrated cross-modal stimulation in wild-caught túngara frogs, a species with well-studied acoustic preferences in females. We first identified that a combined visual and seismic cue (vocal sac movement and water ripple) was behaviourally relevant for females choosing between two courtship calls in a phonotaxis assay. We then found that this combined cross-modal stimulus rescued a species-typical acoustic preference in the presence of background noise that otherwise abolished the preference. These results highlight how cross-modal stimulation can prime attention in receivers to improve performance during decision-making. With this, we provide the foundation for future work uncovering the processes and conditions that promote cross-modal facilitation effects.

Cycling multisensory changes in migraine: more than a headache

PURPOSE OF REVIEW

Research on migraine usually focuses on the headache; however, accumulating evidence suggests that migraine not only changes the somatosensory system for nociception (pain), but also the other modalities of perception, such as visual, auditory or tactile sense. More importantly, the multisensory changes exist beyond the headache (ictal) phase of migraine and show cyclic changes, suggesting a central generator driving the multiple sensory changes across different migraine phases. This review summarizes the latest studies that explored the cyclic sensory changes of migraine.

RECENT FINDINGS

Considerable evidence from recent neurophysiological and functional imaging studies suggests that alterations in brain activation start at least 48 h before the migraine headache and outlast the pain itself for 24 h. Several sensory modalities are involved with cyclic changes in sensitivity that peak during the ictal phase.

SUMMARY

In many ways, migraine represents more than just vascular-mediated headaches. Migraine alters the propagation of sensory information long before the headache attack starts.

The “Primitive Brain Dysfunction” Theory of Autism: The Superior Colliculus Role

A better understanding of the pathogenesis of autism will help clarify our conception of the complexity of normal brain development. The crucial deficit may lie in the postnatal changes that vision produces in the brainstem nuclei during early life. The superior colliculus is the primary brainstem visual center. Although difficult to examine in humans with present techniques, it is known to support behaviors essential for every vertebrate to survive, such as the ability to pay attention to relevant stimuli and to produce automatic motor responses based on sensory input. From birth to death, it acts as a brain sentinel that influences basic aspects of our behavior. It is the main brainstem hub that lies between the environment and the rest of the higher neural system, making continuous, implicit decisions about where to direct our attention. The conserved cortex-like organization of the superior colliculus in all vertebrates allows the early appearance of primitive emotionally-related behaviors essential for survival. It contains first-line specialized neurons enabling the detection and tracking of faces and movements from birth. During development, it also sends the appropriate impulses to help shape brain areas necessary for social-communicative abilities. These abilities require the analysis of numerous variables, such as the simultaneous evaluation of incoming information sustained by separate brain networks (visual, auditory and sensory-motor, social, emotional, etc.), and predictive capabilities which compare present events to previous experiences and possible responses. These critical aspects of decision-making allow us to evaluate the impact that our response or behavior may provoke in others. The purpose of this review is to show that several enigmas about the complexity of autism might be explained by disruptions of collicular and brainstem functions. The results of two separate lines of investigation: 1. the cognitive, etiologic, and pathogenic aspects of autism on one hand, and two. the functional anatomy of the colliculus on the other, are considered in order to bridge the gap between basic brain science and clinical studies and to promote future research in this unexplored area.

Information in Explaining Cognition: How to Evaluate It?

The claims that “The brain processes information” or “Cognition is information processing” are accepted as truisms in cognitive science. However, it is unclear how to evaluate such claims absent a specification of “information” as it is used by neurocognitive theories. The aim of this article is, thus, to identify the key features of information that information-based neurocognitive theories posit. A systematic identification of these features can reveal the explanatory role that information plays in specific neurocognitive theories, and can, therefore, be both theoretically and practically important. These features can be used, in turn, as desiderata against which candidate theories of information may be evaluated. After discussing some characteristics of explanation in cognitive science and their implications for “information”, three notions are briefly introduced: natural, sensory, and endogenous information. Subsequently, six desiderata are identified and defended based on cognitive scientific practices. The global workspace theory of consciousness is then used as a specific case study that arguably posits either five or six corresponding features of information.

Pedagogical Strategies for the Enhancement of Medical Education

Clearly, memory and learning are essential to medical education. To make memory and learning more robust and long-term, educators should turn to the advances in neuroscience and cognitive science to direct their efforts. This paper describes the memory pathways and stages with emphasis leading to long-term memory storage. Particular stress is placed on this storage as a construct known as schema. Leading from this background, several pedagogical strategies are described: cognitive load, dual encoding, spiral syllabus, bridging and chunking, sleep consolidation, and retrieval practice.

Retrieval Practice for Improving Long-Term Retention in Anatomical Education: A Quasi-Experimental Study

It is generally assumed by students that learning takes place during repeated episodes of rereading and rote memorization of course materials. Over the past few decades, however, research has increasingly indicated that the said notion can and should be enhanced with learning paradigms such as retrieval practice (RP). RP occurs when students practice retrieving their consolidated semantic memories by informally testing themselves. This strategy results in the re-encoding and re-consolidation of existing semantic memories, thus strengthening their schemas. The purpose of this quasi-experimental design was to assess the effects of the implementation of RP on student performance on the final exam in a large, undergraduate Gross Anatomy course. It was hypothesized that student participation in RP during class would improve their performance on the final exam in the course. The participants (N = 248) were mainly in Life Sciences, Kinesiology, and Physical Education programs. They answered RP questions using TopHat©, an online educational software platform. The results of this study indicated that student performance on the final exam was enhanced when students engaged in RP. It was concluded that the use of RP effectively enhances learning and long-term retention of semantic memory. In addition to the traditional testing 'of' learning, teachers are encouraged to implement testing, in the form of RP, in their classrooms 'for' learning.

The Relationship Between Multisensory Temporal Processing and Schizotypal Traits

Recent literature has suggested that deficits in sensory processing are associated with schizophrenia (SCZ), and more specifically hallucination severity. The DSM-5's shift towards a dimensional approach to diagnostic criteria has led to SCZ and schizotypal personality disorder (SPD) being classified as schizophrenia spectrum disorders. With SCZ and SPD overlapping in aetiology and symptomatology, such as sensory abnormalities, it is important to investigate whether these deficits commonly reported in SCZ extend to non-clinical expressions of SPD. In this study, we investigated whether levels of SPD traits were related to audiovisual multisensory temporal processing in a non-clinical sample, revealing two novel findings. First, less precise multisensory temporal processing was related to higher overall levels of SPD symptomatology. Second, this relationship was specific to the cognitive-perceptual domain of SPD symptomatology, and more specifically, the Unusual Perceptual Experiences and Odd Beliefs or Magical Thinking symptomatology. The current study provides an initial look at the relationship between multisensory temporal processing and schizotypal traits. Additionally, it builds on the previous literature by suggesting that less precise multisensory temporal processing is not exclusive to SCZ but may also be related to non-clinical expressions of schizotypal traits in the general population.

Visuo-haptic transfer for object recognition in children with peripheral visual impairment

It is well known how early visual experience is critical for the development of multisensory processing abilities, and for this reason an early vision impairment could hinder the transfer of different sensory information during the exploration and recognition of the surrounding environment. Recently, we verified that visuo-haptic transfer for object recognition emerges early in typically developing children but matures slowly during the school-age period. Subsequently we verified the presence of a slower trend of development in unisensory and multisensory skills in children with early abnormal motor and sensory experiences due to brain lesions. Now, we investigated unimodal visual information, unimodal haptic information and visuo-haptic information transfer in children with a diagnosis of low-vision, due to congenital visual impairment. Unimodal and bimodal processes for object recognition were explored in 11 children with low-vision and the results were matched with those of 22 controls. Participants were tested using a clinical protocol involving visual exploration of black-and-white photographs of common objects, haptic exploration of real objects and visuo-haptic transfer of these two types of information. Results show a normal development in haptic unisensory processing in children with low vision and a significant difference in multisensory transfer between the two groups. In children with visual impairment, multisensory processes do not facilitate the recognition of common objects as in typical children, probably because early visual impairment may impact the cross-sensory calibration of vision and touch.

Multisensory Integration of Emotion in Schizophrenic Patients

Multisensory integration (MSI) of emotion has been increasingly recognized as an essential element of schizophrenic patients' impairments, leading to the breakdown of their interpersonal functioning. The present review provides an updated synopsis of schizophrenics' MSI abilities in emotion processing by examining relevant behavioral and neurological research. Existing behavioral studies have adopted well-established experimental paradigms to investigate how participants understand multisensory emotion stimuli, and interpret their reciprocal interactions. Yet it remains controversial with regard to congruence-induced facilitation effects, modality dominance effects, and generalized vs specific impairment hypotheses. Such inconsistencies are likely due to differences and variations in experimental manipulations, participants' clinical symptomatology, and cognitive abilities. Recent electrophysiological and neuroimaging research has revealed aberrant indices in event-related potential (ERP) and brain activation patterns, further suggesting impaired temporal processing and dysfunctional brain regions, connectivity and circuities at different stages of MSI in emotion processing. The limitations of existing studies and implications for future MSI work are discussed in light of research designs and techniques, study samples and stimuli, and clinical applications.

Multimodal machine translation through visuals and speech

Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data. The most prominent tasks in this area are spoken language translation, image-guided translation, and video-guided translation, which exploit audio and visual modalities, respectively. These tasks are distinguished from their monolingual counterparts of speech recognition, image captioning, and video captioning by the requirement of models to generate outputs in a different language. This survey reviews the major data resources for these tasks, the evaluation campaigns concentrated around them, the state of the art in end-to-end and pipeline approaches, and also the challenges in performance evaluation. The paper concludes with a discussion of directions for future research in these areas: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

NMDA Receptors in Visual and Olfactory Sensory Integration in Male Long Evans Rats: A Role for the Orbitofrontal Cortex

Sensory integration (SI) is a cognitive process whereby the brain uses unimodal or multimodal sensory features to create a comprehensive representation of the environment. Integration of sensory input is necessary to achieve a coherent perception of the environment, and to subsequently plan and coordinate action. The neural mechanisms mediating SI are poorly understood; however, recent studies suggest that the regulation of SI involves N-methyl-d-aspartate receptors (NMDARs) in orbitofrontal cortex (OFC). Thus, we tested this hypothesis directly in two experiments using object oddity tests that require SI for visual and olfactory stimuli. First, we blocked NMDARs with acute CPP treatment (i.p., 10 mg/kg) and tested rats in unimodal visual and olfactory SI tests, and respective control unimodal oddity tests that do not require SI. Second, we used intra-OFC infusions of AP5 (30 mM) to examine the role of NMDARs in the OFC in the oddity tests requiring SI. Systemic blockade of NMDARs impaired performance on the visual tests regardless of whether SI was required for determining oddity. In the olfactory tests, systemic treatment with CPP impaired the test requiring SI while sparing olfactory oddity, demonstrating a selective impairment in the olfactory SI. Intra-OFC blockade of NMDARs impaired olfactory SI, without effect on visual SI, demonstrating that intra-OFC NMDARs are essential for olfactory, but not visual SI. The present results are discussed in the context of the function of the OFC and its associated circuitry.

Efficiency of Sensory Substitution Devices Alone and in Combination With Self-Motion for Spatial Navigation in Sighted and Visually Impaired

Human adults can optimally combine vision with self-motion to facilitate navigation. In the absence of visual input (e.g., dark environments and visual impairments), sensory substitution devices (SSDs), such as The vOICe or BrainPort, which translate visual information into auditory or tactile information, could be used to increase navigation precision when integrated together or with self-motion. In Experiment 1, we compared and assessed together The vOICe and BrainPort in aerial maps task performed by a group of sighted participants. In Experiment 2, we examined whether sighted individuals and a group of visually impaired (VI) individuals could benefit from using The vOICe, with and without self-motion, to accurately navigate a three-dimensional (3D) environment. In both studies, 3D motion tracking data were used to determine the level of precision with which participants performed two different tasks (an egocentric and an allocentric task) and three different conditions (two unisensory conditions and one multisensory condition). In Experiment 1, we found no benefit of using the devices together. In Experiment 2, the sighted performance during The vOICe was almost as good as that for self-motion despite a short training period, although we found no benefit (reduction in variability) of using The vOICe and self-motion in combination compared to the two in isolation. In contrast, the group of VI participants did benefit from combining The vOICe and self-motion despite the low number of trials. Finally, while both groups became more accurate in their use of The vOICe with increased trials, only the VI group showed an increased level of accuracy in the combined condition. Our findings highlight how exploiting non-visual multisensory integration to develop new assistive technologies could be key to help blind and VI persons, especially due to their difficulty in attaining allocentric information.

Problems in Audiovisual Filtering for Children with Special Educational Needs

There is pervasive evidence that problems in sensory processing occur across a range of developmental disorders, but their aetiology and clinical significance remain unclear. The present study investigated the relation between sensory processing and literacy skills in children with and without a background of special educational needs (SEN). Twenty-six children aged between 7 and 12 years old, from both regular classes and SEN programmes, participated. Following baseline tests of literacy, fine motor skills and naming speed, two sets of instruments were administered: the carer-assessed Child Sensory Profile-2 and a novel Audiovisual Animal Stroop (AVAS) test. The SEN group showed significantly higher ratings on three Child Sensory Profile-2 quadrants, together with body position ratings. The SEN participants also showed a specific deficit when required to ignore an accompanying incongruent auditory stimulus on the AVAS. Interestingly, AVAS performance correlated significantly with literacy scores and with the sensory profile scores. It is proposed that the children with SEN showed a specific deficit in "filtering out" irrelevant auditory input. The results highlight the importance of including analysis of sensory processes within theoretical and applied approaches to developmental differences and suggest promising new approaches to the understanding, assessment, and support of children with SEN.

Conscious Multisensory Integration: Introducing a Universal Contextual Field in Biological and Deep Artificial Neural Networks

Conscious awareness plays a major role in human cognition and adaptive behavior, though its function in multisensory integration is not yet fully understood, hence, questions remain: How does the brain integrate the incoming multisensory signals with respect to different external environments? How are the roles of these multisensory signals defined to adhere to the anticipated behavioral-constraint of the environment? This work seeks to articulate a novel theory on conscious multisensory integration (CMI) that addresses the aforementioned research challenges. Specifically, the well-established contextual field (CF) in pyramidal cells and coherent infomax theory (Kay et al., 1998; Kay and Phillips, 2011) is split into two functionally distinctive integrated input fields: local contextual field (LCF) and universal contextual field (UCF). LCF defines the modulatory sensory signal coming from some other parts of the brain (in principle from anywhere in space-time) and UCF defines the outside environment and anticipated behavior (based on past learning and reasoning). Both LCF and UCF are integrated with the receptive field (RF) to develop a new class of contextually-adaptive neuron (CAN), which adapts to changing environments. The proposed theory is evaluated using human contextual audio-visual (AV) speech modeling. Simulation results provide new insights into contextual modulation and selective multisensory information amplification/suppression. The central hypothesis reviewed here suggests that the pyramidal cell, in addition to the classical excitatory and inhibitory signals, receives LCF and UCF inputs. The UCF (as a steering force or tuner) plays a decisive role in precisely selecting whether to amplify/suppress the transmission of relevant/irrelevant feedforward signals, without changing the content e.g., which information is worth paying more attention to? This, as opposed to, unconditional excitatory and inhibitory activity in existing deep neural networks (DNNs), is called conditional amplification/suppression.

Processing of Real-World, Dynamic Natural Stimuli in Autism is Linked to Corticobasal Function

Many individuals with autism spectrum disorder (ASD) have been shown to perceive everyday sensory information differently compared to peers without autism. Research examining these sensory differences has primarily utilized nonnatural stimuli or natural stimuli using static photos with few having utilized dynamic, real-world nonverbal stimuli. Therefore, in this study, we used functional magnetic resonance imaging to characterize brain activation of individuals with high-functioning autism when viewing and listening to a video of a real-world scene (a person bouncing a ball) and anticipating the bounce. We investigated both multisensory and unisensory processing and hypothesized that individuals with ASD would show differential activation in (a) primary auditory and visual sensory cortical and association areas, and in (b) cortical and subcortical regions where auditory and visual information is integrated (e.g. temporal-parietal junction, pulvinar, superior colliculus). Contrary to our hypotheses, the whole-brain analysis revealed similar activation between the groups in these brain regions. However, compared to controls the ASD group showed significant hypoactivation in the left intraparietal sulcus and left putamen/globus pallidus. We theorize that this hypoactivation reflected underconnectivity for mediating spatiotemporal processing of the visual biological motion stimuli with the task demands of anticipating the timing of the bounce event. The paradigm thus may have tapped into a specific left-lateralized aberrant corticobasal circuit or loop involved in initiating or inhibiting motor responses. This was consistent with a dual "when versus where" psychophysical model of corticobasal function, which may reflect core differences in sensory processing of real-world, nonverbal natural stimuli in ASD. Autism Res 2020, 13: 539-549. © 2020 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: To understand how individuals with autism perceive the real-world, using magnetic resonance imaging we examined brain activation in individuals with autism while watching a video of someone bouncing a basketball. Those with autism had similar activation to controls in auditory and visual sensory brain regions, but less activation in an area that processes information about body movements and in a region involved in modulating movements. These areas are important for understanding the actions of others and developing social skills.

Enhancing Physical Human Evasion of Moving Threats Using Tactile Cues

New human-centric approaches to safety can combine over-head camera views with situational awareness tools to enable humans to avoid rapidly evolving threats such as moving machines or falling debris. This article explores how 360° information can be used to inform humans of potential collisions. Specifically, we quantify how different individual (tactile, audio, and visual) and combined cue modalities affect failure rates and reaction times. Human-subject experiments were conducted in a custom virtual reality environment that simulates objects rapidly moving toward the subject. In order to successfully perform their task, the human subject must physically move their body out of the path of the moving threat before a collision occurs. This exploration of full body physical response differentiates this article from previous related studies. The results of the 18-subject study provide quantified data on a range of cues and cue combinations. The study quantified failure rates and reaction times as a function of index of difficulty (Fitt's Law) and threat directionality. The results confirm the hypothesis that the addition of tactile cues statistically improve performance compared to non-tactile cues with regards to failure rate and reaction time. This demonstrates how sensory cues can improve human physical response to rapid threats.

A VTA GABAergic Neural Circuit Mediates Visually Evoked Innate Defensive Responses

Innate defensive responses are essential for animal survival and are conserved across species. The ventral tegmental area (VTA) plays important roles in learned appetitive and aversive behaviors, but whether it plays a role in mediating or modulating innate defensive responses is currently unknown. We report that VTA^GABA+ neurons respond to a looming stimulus. Inhibition of VTA^GABA+ neurons reduced looming-evoked defensive flight behavior, and photoactivation of these neurons resulted in defense-like flight behavior. Using viral tracing and electrophysiological recordings, we show that VTA^GABA+ neurons receive direct excitatory inputs from the superior colliculus (SC). Furthermore, we show that glutamatergic SC-VTA projections synapse onto VTA^GABA+ neurons that project to the central nucleus of the amygdala (CeA) and that the CeA is involved in mediating the defensive behavior. Our findings demonstrate that aerial threat-related visual information is relayed to VTA^GABA+ neurons mediating innate behavioral responses, suggesting a more general role of the VTA.

MRI acoustic noise-modulated computer animations for patient distraction and entertainment with application in pediatric psychiatric patients

PURPOSE

To reduce patient anxiety caused by the MRI scanner acoustic noise.

MATERIAL AND METHODS

We developed a simple and low-cost system for patient distraction using visual computer animations that were synchronized to the MRI scanner's acoustic noise during the MRI exam. The system was implemented on a 3T MRI system and tested in 28 pediatric patients with bipolar disorder. The patients were randomized to receive noise-synchronized animations in the form of abstract animations in addition to music (n = 13, F/M = 6/7, age = 10.9 ± 2.5 years) or, as a control, receive only music (n = 15, F/M = 7/8, age = 11.6 ± 2.3 years). After completion of the scans, all subjects answered a questionnaire about their scan experience and the perceived scan duration.

RESULTS

The scan duration with multisensory input (animations and music) was perceived to be ~15% shorter than in the control group (43 min vs. 50 min, P < 0.05). However, the overall scan experience was scored less favorably (3.9 vs. 4.6 in the control group, P < 0.04).

CONCLUSIONS

This simple system provided patient distraction and entertainment leading to perceived shorter scan times, but the provided visualization with abstract animations was not favored by this patient cohort.

Sensus Communis: Some Perspectives on the Origins of Non-synchronous Cross-Sensory Associations

Adults readily make associations between stimuli perceived consecutively through different sense modalities, such as shapes and sounds. Researchers have only recently begun to investigate such correspondences in infants but only a handful of studies have focused on infants less than a year old. Are infants able to make cross-sensory correspondences from birth? Do certain correspondences require extensive real-world experience? Some studies have shown that newborns are able to match stimuli perceived in different sense modalities. Yet, the origins and mechanisms underlying these abilities are unclear. The present paper explores these questions and reviews some hypotheses on the emergence and early development of cross-sensory associations and their possible links with language development. Indeed, if infants can perceive cross-sensory correspondences between events that share certain features but are not strictly contingent or co-located, one may posit that they are using a "sixth sense" in Aristotle's sense of the term. And a likely candidate for explaining this mechanism, as Aristotle suggested, is movement.

The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network

Our perceptual systems continuously process sensory inputs from different modalities and organize these streams of information such that our subjective representation of the outside world is a unified experience. By doing so, they also enable further cognitive processing and behavioral action. While cortical multisensory processing has been extensively investigated in terms of psychophysics and mesoscale neural correlates, an in depth understanding of the underlying circuit-level mechanisms is lacking. Previous studies on circuit-level mechanisms of multisensory processing have predominantly focused on cue integration, i.e. the mechanism by which sensory features from different modalities are combined to yield more reliable stimulus estimates than those obtained by using single sensory modalities. In this review, we expand the framework on the circuit-level mechanisms of cortical multisensory processing by highlighting that multisensory processing is a family of functions - rather than a single operation - which involves not only the integration but also the segregation of modalities. In addition, multisensory processing not only depends on stimulus features, but also on cognitive resources, such as attention and memory, as well as behavioral context, to determine the behavioral outcome. We focus on rodent models as a powerful instrument to study the circuit-level bases of multisensory processes, because they enable combining cell-type-specific recording and interventional techniques with complex behavioral paradigms. We conclude that distinct multisensory processes share overlapping anatomical substrates, are implemented by diverse neuronal micro-circuitries that operate in parallel, and are flexibly recruited based on factors such as stimulus features and behavioral constraints.

Autism Pathogenesis: The Superior Colliculus

After been exposed to the visual input, in the first year of life, the brain experiences subtle but massive changes apparently crucial for communicative/emotional and social human development. Its lack could be the explanation of the very high prevalence of autism in children with total congenital blindness. The present theory postulates that the superior colliculus is the key structure for such changes for several reasons: it dominates visual behavior during the first months of life; it is ready at birth for complex visual tasks; it has a significant influence on several hemispheric regions; it is the main brain hub that permanently integrates visual and non-visual, external and internal information (bottom-up and top-down respectively); and it owns the enigmatic ability to take non-conscious decisions about where to focus attention. It is also a sentinel that triggers the subcortical mechanisms which drive social motivation to follow faces from birth and to react automatically to emotional stimuli. Through indirect connections it also activates simultaneously several cortical structures necessary to develop social cognition and to accomplish the multiattentional task required for conscious social interaction in real life settings. Genetic or non-genetic prenatal or early postnatal factors could disrupt the SC functions resulting in autism. The timing of postnatal biological disruption matches the timing of clinical autism manifestations. Astonishing coincidences between etiologies, clinical manifestations, cognitive and pathogenic autism theories on one side and SC functions on the other are disclosed in this review. Although the visual system dependent of the SC is usually considered as accessory of the LGN canonical pathway, its imprinting gives the brain a qualitatively specific functions not supplied by any other brain structure.

Audiovisual multisensory integration in individuals with autism spectrum disorder: A systematic review and meta-analysis

An ever-growing literature has aimed to determine how individuals with autism spectrum disorder (ASD) differ from their typically developing (TD) peers on measures of multisensory integration (MSI) and to ascertain the degree to which differences in MSI are associated with the broad range of symptoms associated with ASD. Findings, however, have been highly variable across the studies carried out to date. The present work systematically reviews and quantitatively synthesizes the large literature on audiovisual MSI in individuals with ASD to evaluate the cumulative evidence for (a) group differences between individuals with ASD and TD peers, (b) correlations between MSI and autism symptoms in individuals with ASD and (c) study level factors that may moderate findings (i.e., explain differential effects) observed across studies. To identify eligible studies, a comprehensive search strategy was employed using the ProQuest search engine, PubMed database, forwards and backwards citation searches, direct author contact, and hand-searching of select conference proceedings. A significant between-group difference in MSI was evident in the literature, with individuals with ASD demonstrating worse audiovisual integration on average across studies compared to TD controls. This effect was moderated by mean participant age, such that between-group differences were more pronounced in younger samples. The mean correlation between MSI and autism and related symptomatology was also significant, indicating that increased audiovisual integration in individuals with ASD is associated with better language/communication abilities and/or reduced autism symptom severity in the extant literature. This effect was moderated by whether the stimuli were linguistic versus non-linguistic in nature, such that correlation magnitudes tended to be significantly greater when linguistic stimuli were utilized in the measure of MSI. Limitations and future directions for primary and meta-analytic research are discussed.

Pulvinar Lesions Disrupt Fear-Related Implicit Visual Processing in Hemianopic Patients

The processing of emotional stimuli in the absence of awareness has been widely investigated in patients with lesions to the primary visual pathway since the classical studies on affective blindsight. In addition, recent evidence has shown that in hemianopic patients without blindsight only unseen fearful faces can be implicitly processed, inducing enhanced visual encoding (Cecere et al., 2014) and response facilitation (Bertini et al., 2013, 2017) to stimuli presented in their intact field. This fear-specific facilitation has been suggested to be mediated by activity in the spared visual subcortical pathway, comprising the superior colliculus (SC), the pulvinar and the amygdala. This suggests that the pulvinar might represent a critical relay structure, conveying threat-related visual information through the subcortical visual circuit. To test this hypothesis, hemianopic patients, with or without pulvinar lesions, performed a go/no-go task in which they had to discriminate simple visual stimuli, consisting in Gabor patches, displayed in their intact visual field, during the simultaneous presentation of faces with fearful, happy, and neutral expressions in their blind visual field. In line with previous evidence, hemianopic patients without pulvinar lesions showed response facilitation to stimuli displayed in the intact field, only while concurrent fearful faces were shown in their blind field. In contrast, no facilitatory effect was found in hemianopic patients with lesions of the pulvinar. These findings reveal that pulvinar lesions disrupt the implicit visual processing of fearful stimuli in hemianopic patients, therefore suggesting a pivotal role of this structure in relaying fear-related visual information from the SC to the amygdala.

Colocalization of Tectal Inputs With Amygdala-Projecting Neurons in the Macaque Pulvinar

Neuropsychological and neuroimaging studies have suggested the presence of a fast, subcortical route for the processing of emotionally-salient visual information in the primate brain. This putative pathway consists of the superior colliculus (SC), pulvinar and amygdala. While the presence of such a pathway has been confirmed in sub-primate species, it has yet to be documented in the primate brain using conventional anatomical methods. We injected retrograde tracers into the amygdala and anterograde tracers into the colliculus, and examined regions of colocalization of these signals within the pulvinar of the macaque. Anterograde tracers injected into the SC labeled axonal projections within the pulvinar, primarily within the oral, lateral and medial subdivisions. These axonal projections from the colliculus colocalized with cell bodies within the pulvinar that were labeled by retrograde tracer injected into the lateral amygdala. This zone of overlap was most notable in the medial portions of the medial (PM), oral (PO) and inferior pulvinar (PI), and was often densely concentrated in the vicinity of the brachium of the SC. These data provide an anatomical basis for the previously suggested pathway mediating fast processing of emotionally salient information.

Spatial receptive field shift by preceding cross-modal stimulation in the cat superior colliculus

KEY POINTS

It has been known for some time that sensory information of one type can bias the spatial perception of another modality. However, there is a lack of evidence of this occurring in individual neurons. In the present study, we found that the spatial receptive field of superior colliculus multisensory neurons could be dynamically shifted by a preceding stimulus in a different modality. The extent to which the receptive field shifted was dependent on both temporal and spatial gaps between the preceding and following stimuli, as well as the salience of the preceding stimulus. This result provides a neural mechanism that could underlie the process of cross-modal spatial calibration.

ABSTRACT

Psychophysical studies have shown that the different senses can be spatially entrained by each other. This can be observed in certain phenomena, such as ventriloquism, in which a visual stimulus can attract the perceived location of a spatially discordant sound. However, the neural mechanism underlying this cross-modal spatial recalibration has remained unclear, as has whether it takes place dynamically. We explored these issues in multisensory neurons of the cat superior colliculus (SC), a midbrain structure that involves both cross-modal and sensorimotor integration. Sequential cross-modal stimulation showed that the preceding stimulus can shift the receptive field (RF) of the lagging response. This cross-modal spatial calibration took place in both auditory and visual RFs, although auditory RFs shifted slightly more. By contrast, if a preceding stimulus was from the same modality, it failed to induce a similarly substantial RF shift. The extent of the RF shift was dependent on both temporal and spatial gaps between the preceding and following stimuli, as well as the salience of the preceding stimulus. A narrow time gap and high stimulus salience were able to induce larger RF shifts. In addition, when both visual and auditory stimuli were presented simultaneously, a substantial RF shift toward the location-fixed stimulus was also induced. These results, taken together, reveal an online cross-modal process and reflect the details of the organization of SC inter-sensory spatial calibration.

The Psychophysiology of Action: A Multidisciplinary Endeavor for Integrating Action and Cognition

There is a vast amount of literature concerning the integration of action and cognition. Although this broad research area is of great interest for many disciplines like sports, psychology and cognitive neuroscience, only a few attempts tried to bring together different perspectives so far. Our goal is to provide a perspective to spark a debate across theoretical borders and integration of different disciplines via psychophysiology. In order to boost advances in this research field it is not only necessary to become aware of the different areas that are relevant but also to consider methodological aspects and challenges. We briefly describe the most relevant theoretical accounts to the question of how internal and external information processes or factors interact and, based on this, argue that research programs should consider the three dimensions: (a) dynamics of movements; (b) multivariate measures and; (c) dynamic statistical parameters. Only with an extended perspective on theoretical and methodological accounts, one would be able to integrate the dynamics of actions into theoretical advances.

Augmenting EEG-global-coherence with auditory and visual noise: Multisensory internal stochastic resonance

The present investigation documents the electrophysiological occurrence of multisensory internal stochastic resonance (MISR) in the human electroencephalographic (EEG) coherence elicited by auditory and visual noise.We define MISR of EEG coherence as the phenomenon for which an intermediate level of input noise of a sensory modality enhances EEG coherence in response to another noisy sensory modality. Here, EEG coherence is computed by the global weighted coherence (GWC), modulated by quasi-Brownian noise. Specifically, we examined whether a particular level of auditory noise together with constant visual noise (experimental condition 1) and a specified level of visual noise together with constant auditory noise (experimental condition 2), improves EEG's GWC. We compared GWC between ongoing EEG basal activity (BA), zero noise (ZN), optimal noise (ON), and high noise (HN).The data disclosed an intermediate level of input noise that enhances the GWC for the majority of the subjects, thus demonstrating for the first time the occurrence of multisensory internal stochastic resonance (SR) in visuoauditory processing within the central nervous system.

Design and Evaluation of a Collaborative Virtual Environment (CoMove) for Autism Spectrum Disorder Intervention

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized in part by core deficits in social interaction and communication. A collaborative virtual environment (CVE), which is a computer-based, distributed, virtual space for multiple users to interact with one another and/or with virtual items, has the potential to support flexible, safe, and peer-based social interactions. In this article, we presented the design of a CVE system, called CoMove, with the ultimate goals of measuring and potentially enhancing collaborative interactions and verbal communication of children with ASD when they play collaborative puzzle games with their typically developing (TD) peers in remote locations. CoMove has two distinguishing characteristics: (i) the ability to promote important collaborative behaviors (including information sharing, sequential interactions, and simultaneous interactions) and to provide real-time feedback based on users’ game performance; as well as (ii) an objective way to measure and index important aspects of collaboration and verbal-communication skills during system interaction. A feasibility study with 14 pairs—7 ASD/TD pairs and 7 TD/TD pairs—was conducted to initially test the feasibility of CoMove. The results of the study validated the system feasibility and suggested its potential to index important aspects of collaboration and verbal communication.

FoxP in bees: A comparative study on the developmental and adult expression pattern in three bee species considering isoforms and circuitry

Mutations in the transcription factors FOXP1, FOXP2, and FOXP4 affect human cognition, including language. The FoxP gene locus is evolutionarily ancient and highly conserved in its DNA-binding domain. In Drosophila melanogaster FoxP has been implicated in courtship behavior, decision making, and specific types of motor-learning. Because honeybees (Apis mellifera, Am) excel at navigation and symbolic dance communication, they are a particularly suitable insect species to investigate a potential link between neural FoxP expression and cognition. We characterized two AmFoxP isoforms and mapped their expression in the brain during development and in adult foragers. Using a custom-made antiserum and in situ hybridization, we describe 11 AmFoxP expressing neuron populations. FoxP was expressed in equivalent patterns in two other representatives of Apidae; a closely related dwarf bee and a bumblebee species. Neural tracing revealed that the largest FoxP expressing neuron cluster in honeybees projects into a posterior tract that connects the optic lobe to the posterior lateral protocerebrum, predicting a function in visual processing. Our data provide an entry point for future experiments assessing the function of FoxP in eusocial Hymenoptera.

Multisensory-Based Rehabilitation Approach: Translational Insights from Animal Models to Early Intervention

Multisensory processes permit combinations of several inputs, coming from different sensory systems, allowing for a coherent representation of biological events and facilitating adaptation to environment. For these reasons, their application in neurological and neuropsychological rehabilitation has been enhanced in the last decades. Recent studies on animals and human models have indicated that, on one hand multisensory integration matures gradually during post-natal life and development is closely linked to environment and experience and, on the other hand, that modality-specific information seems to do not benefit by redundancy across multiple sense modalities and is more readily perceived in unimodal than in multimodal stimulation. In this review, multisensory process development is analyzed, highlighting clinical effects in animal and human models of its manipulation for rehabilitation of sensory disorders. In addition, new methods of early intervention based on multisensory-based rehabilitation approach and their applications on different infant populations at risk of neurodevelopmental disabilities are discussed.

Multiassociative Memory: Recurrent Synapses Increase Storage Capacity

The connection density of nearby neurons in the cortex has been observed to be around 0.1, whereas the longer-range connections are present with much sparser density (Kalisman, Silberberg, & Markram, 2005 ). We propose a memory association model that qualitatively explains these empirical observations. The model we consider is a multiassociative, sparse, Willshaw-like model consisting of binary threshold neurons and binary synapses. It uses recurrent synapses for iterative retrieval of stored memories. We quantify the usefulness of recurrent synapses by simulating the model for small network sizes and by doing a precise mathematical analysis for large network sizes. Given the network parameters, we can determine the precise values of recurrent and afferent synapse densities that optimize the storage capacity of the network. If the network size is like that of a cortical column, then the predicted optimal recurrent density lies in a range that is compatible with biological measurements. Furthermore, we show that our model is able to surpass the standard Willshaw model in the multiassociative case if the information capacity is normalized per strong synapse or per bits required to store the model, as considered in Knoblauch, Palm, and Sommer ( 2010 ).