Convergent approaches toward the study of multisensory perception

Separation of bimodal fMRI responses in mouse somatosensory areas into V1 and non-V1 contributions

Multisensory integration is necessary for the animal to survive in the real world. While conventional methods have been extensively used to investigate the multisensory integration process in various brain areas, its long-range interactions remain less explored. In this study, our goal was to investigate interactions between visual and somatosensory networks on a whole-brain scale using 15.2-T BOLD fMRI. We compared unimodal to bimodal BOLD fMRI responses and dissected potential cross-modal pathways with silencing of primary visual cortex (V1) by optogenetic stimulation of local GABAergic neurons. Our data showed that the influence of visual stimulus on whisker activity is higher than the influence of whisker stimulus on visual activity. Optogenetic silencing of V1 revealed that visual information is conveyed to whisker processing via both V1 and non-V1 pathways. The first-order ventral posteromedial thalamic nucleus (VPM) was functionally affected by non-V1 sources, while the higher-order posterior medial thalamic nucleus (POm) was predominantly modulated by V1 but not non-V1 inputs. The primary somatosensory barrel field (S1BF) was influenced by both V1 and non-V1 inputs. These observations provide valuable insights for into the integration of whisker and visual sensory information.

Deafness induces complete crossmodal plasticity in a belt region of dorsal auditory cortex

Many neural areas, where patterned activity is lost following deafness, have the capacity to become activated by the remaining sensory systems. This crossmodal plasticity can be measured at perceptual/behavioural as well as physiological levels. The dorsal zone (DZ) of auditory cortex of deaf cats is involved in supranormal visual motion detection, but its physiological level of crossmodal reorganisation is not well understood. The present study of early-deaf DZ (and hearing controls) used multiple single-channel recording methods to examine neuronal responses to visual, auditory, somatosensory and combined stimulation. In early-deaf DZ, no auditory activation was observed, but 100% of the neurons were responsive to visual cues of which 21% were also influenced by somatosensory stimulation. Visual and somatosensory responses were not anatomically organised as they are in hearing cats, and fewer multisensory neurons were present in the deaf condition. These crossmodal physiological results closely correspond with and support the perceptual/behavioural enhancements that occur following hearing loss.

MULTISENSORY RESPONSES IN A BELT REGION OF THE DORSAL AUDITORY CORTICAL PATHWAY

A basic function of the cerebral cortex is to receive and integrate information from different sensory modalities into a comprehensive percept of the environment. Neurons that demonstrate multisensory convergence occur across the necortex, but are especially prevalent in higher-order, association areas. However, a recent study of a cat higher-order auditory area, the dorsal zone (DZ) of auditory cortex, did not observe any multisensory features. Therefore, the goal of the present investigation was to address this conflict using recording and testing methodologies that are established for exposing and studying multisensory neuronal processing. Among the 482 neurons studied, we found that 76.6% were influenced by non-auditory stimuli. Of these neurons, 99% were affected by visual stimulation, but only 11% by somatosensory. Furthermore, a large proportion of the multisensory neurons showed integrated responses to multisensory stimulation, constituted a majority of the excitatory and inhibitory neurons encountered (as identified by the duration of their waveshape), and exhibited a distinct spatial distribution within DZ. These findings demonstrate that the dorsal zone of auditory cortex robustly exhibits multisensory properties and that the proportions of multisensory neurons encountered are consistent with those identified in other higher-order cortices.

Developmental plasticity of texture discrimination following early vision loss in the marsupial Monodelphis domestica

Behavioral strategies that depend on sensory information are not immutable; rather they can be shaped by the specific sensory context in which animals develop. This behavioral plasticity depends on the remarkable capacity of the brain to reorganize in response to alterations in the sensory environment, particularly when changes in sensory input occur at an early age. To study this phenomenon, we utilize the short-tailed opossum, a marsupial that has been a valuable animal model to study developmental plasticity due to the extremely immature state of its nervous system at birth. Previous studies in opossums have demonstrated that removal of retinal inputs early in development results in profound alterations to cortical connectivity and functional organization of visual and somatosensory cortex; however, behavioral consequences of this plasticity are not well understood. We trained early blind and sighted control opossums to perform a two-alternative forced choice texture discrimination task. Whisker trimming caused an acute deficit in discrimination accuracy for both groups, indicating the use of a primarily whisker-based strategy to guide choices based on tactile cues. Mystacial whiskers were important for performance in both groups; however, genal whiskers only contributed to behavioral performance in early blind animals. Early blind opossums significantly outperformed their sighted counterparts in discrimination accuracy, with discrimination thresholds that were lower by ∼75 μm. Our results support behavioral compensation following early blindness using tactile inputs, especially the whisker system.

Sensory feedback-dependent coding of arm position in local field potentials of the posterior parietal cortex

Although multisensory integration is crucial for sensorimotor function, it is unclear how visual and proprioceptive sensory cues are combined in the brain during motor behaviors. Here we characterized the effects of multisensory interactions on local field potential (LFP) activity obtained from the superior parietal lobule (SPL) as non-human primates performed a reaching task with either unimodal (proprioceptive) or bimodal (visual-proprioceptive) sensory feedback. Based on previous analyses of spiking activity, we hypothesized that evoked LFP responses would be tuned to arm location but would be suppressed on bimodal trials, relative to unimodal trials. We also expected to see a substantial number of recording sites with enhanced beta band spectral power for only one set of feedback conditions (e.g. unimodal or bimodal), as was previously observed for spiking activity. We found that evoked activity and beta band power were tuned to arm location at many individual sites, though this tuning often differed between unimodal and bimodal trials. Across the population, both evoked and beta activity were consistent with feedback-dependent tuning to arm location, while beta band activity also showed evidence of response suppression on bimodal trials. The results suggest that multisensory interactions can alter the tuning and gain of arm position-related LFP activity in the SPL.

Even distorted felt size engenders visual-haptic integration

Intersensory interactions predicted by the sensory precision hypothesis have been infrequently examined by distorting the reliability of size perception by touch. Consequently, participants were asked to see one size and manually feel another unseen size either with bare fingers or with fingers sleeved in rigid tubes to decrease the precision of touch. Their subsequent visual estimates of the perceived size favored the more precise modality. Experiment 1 (N = 46) varied the intersensory discordance to examine whether the estimate arose from trivial response biases or from perceptual binding effects. Experiment 2 (N = 32) examined the presence of the perceptual effect in the absence of discordant sensory cues. Results favored a perceptual interpretation because the haptic and visual cues merged regardless of the discordance amount only when the stimulation arose from separate sources. The observed interaction between touch imprecision and visual bias is consistent with computational models of optimal perception.

Context-specific modulation of intrinsic coupling modes shapes multisensory processing

Intrinsically generated patterns of coupled neuronal activity are associated with the dynamics of specific brain states. Sensory inputs are extrinsic factors that can perturb these intrinsic coupling modes, creating a complex scenario in which forthcoming stimuli are processed. Studying this intrinsic-extrinsic interplay is necessary to better understand perceptual integration and selection. Here, we show that this interplay leads to a reconfiguration of functional cortical connectivity that acts as a mechanism to facilitate stimulus processing. Using audiovisual stimulation in anesthetized ferrets, we found that this reconfiguration of coupling modes is context specific, depending on long-term modulation by repetitive sensory inputs. These reconfigured coupling modes lead to changes in latencies and power of local field potential responses that support multisensory integration. Our study demonstrates that this interplay extends across multiple time scales and involves different types of intrinsic coupling. These results suggest a previously unknown large-scale mechanism that facilitates multisensory integration.

Multisensory integration in rodent tactile but not visual thalamus

Behavioural performance requires a coherent perception of environmental features that address multiple senses. These diverse sensory inputs are integrated in primary sensory cortices, yet it is still largely unknown whether their convergence occurs even earlier along the sensory tract. Here we investigate the role of putatively modality-specific first-order (FO) thalamic nuclei (ventral posteromedial nucleus (VPM), dorsal lateral geniculate nucleus (dLGN)) and their interactions with primary sensory cortices (S1, V1) for multisensory integration in pigmented rats in vivo. We show that bimodal stimulation (i.e. simultaneous light flash and whisker deflection) enhances sensory evoked activity in VPM, but not dLGN. Moreover, cross-modal stimuli reset the phase of thalamic network oscillations and strengthen the coupling efficiency between VPM and S1, but not between dLGN and V1. Finally, the information flow from VPM to S1 is enhanced. Thus, FO tactile, but not visual, thalamus processes and relays sensory inputs from multiple senses, revealing a functional difference between sensory thalamic nuclei during multisensory integration.

Contributions of Intraindividual and Interindividual Differences to Multisensory Processes

Most evidence on the neural and perceptual correlates of sensory processing derives from studies that have focused on only a single sensory modality and averaged the data from groups of participants. Although valuable, such studies ignore the substantial interindividual and intraindividual differences that are undoubtedly at play. Such variability plays an integral role in both the behavioral/perceptual realms and in the neural correlates of these processes, but substantially less is known when compared with group-averaged data. Recently, it has been shown that the presentation of stimuli from two or more sensory modalities (i.e., multisensory stimulation) not only results in the well-established performance gains but also gives rise to reductions in behavioral and neural response variability. To better understand the relationship between neural and behavioral response variability under multisensory conditions, this study investigated both behavior and brain activity in a task requiring participants to discriminate moving versus static stimuli presented in either a unisensory or multisensory context. EEG data were analyzed with respect to intraindividual and interindividual differences in RTs. The results showed that trial-by-trial variability of RTs was significantly reduced under audiovisual presentation conditions as compared with visual-only presentations across all participants. Intraindividual variability of RTs was linked to changes in correlated activity between clusters within an occipital to frontal network. In addition, interindividual variability of RTs was linked to differential recruitment of medial frontal cortices. The present findings highlight differences in the brain networks that support behavioral benefits during unisensory versus multisensory motion detection and provide an important view into the functional dynamics within neuronal networks underpinning intraindividual performance differences.

Supralinear and Supramodal Integration of Visual and Tactile Signals in Rats: Psychophysics and Neuronal Mechanisms

To better understand how object recognition can be triggered independently of the sensory channel through which information is acquired, we devised a task in which rats judged the orientation of a raised, black and white grating. They learned to recognize two categories of orientation: 0° ± 45° (“horizontal”) and 90° ± 45° (“vertical”). Each trial required a visual (V), a tactile (T), or a visual-tactile (VT) discrimination; VT performance was better than that predicted by optimal linear combination of V and T signals, indicating synergy between sensory channels. We examined posterior parietal cortex (PPC) and uncovered key neuronal correlates of the behavioral findings: PPC carried both graded information about object orientation and categorical information about the rat’s upcoming choice; single neurons exhibited identical responses under the three modality conditions. Finally, a linear classifier of neuronal population firing replicated the behavioral findings. Taken together, these findings suggest that PPC is involved in the supramodal processing of shape.

•

Rats combine vision and touch to distinguish two grating orientation categories

•

Performance with vision and touch together reveals synergy between the two channels

•

Posterior parietal cortex (PPC) neuronal responses are invariant to modality

•

PPC neurons carry information about object orientation and the rat’s categorization

Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots

For children with autism spectrum disorders (ASDs), social robots are increasingly utilized as therapeutic tools in order to enhance social skills and communication. Robots have been shown to generate a number of social and behavioral benefits in children with ASD including heightened engagement, increased attention, and decreased social anxiety. Although social robots appear to be effective social reinforcement tools in assistive therapies, the perceptual mechanism underlying these benefits remains unknown. To date, social robot studies have primarily relied on expertise in fields such as engineering and clinical psychology, with measures of social robot efficacy principally limited to qualitative observational assessments of children's interactions with robots. In this review, we examine a range of socially interactive robots that currently have the most widespread use as well as the utility of these robots and their therapeutic effects. In addition, given that social interactions rely on audiovisual communication, we discuss how enhanced sensory processing and integration of robotic social cues may underlie the perceptual and behavioral benefits that social robots confer. Although overall multisensory processing (including audiovisual integration) is impaired in individuals with ASD, social robot interactions may provide therapeutic benefits by allowing audiovisual social cues to be experienced through a simplified version of a human interaction. By applying systems neuroscience tools to identify, analyze, and extend the multisensory perceptual substrates that may underlie the therapeutic benefits of social robots, future studies have the potential to strengthen the clinical utility of social robots for individuals with ASD.

Serotonergic Modulation of Sensory and Multisensory Processing in Superior Colliculus

The ability to integrate information across the senses is vital for coherent perception of and interaction with the world. While much is known regarding the organization and function of multisensory neurons within the mammalian superior colliculus (SC), very little is understood at a mechanistic level. One open question in this regard is the role of neuromodulatory networks in shaping multisensory responses. While the SC receives substantial serotonergic projections from the raphe nuclei, and serotonergic receptors are distributed throughout the SC, the potential role of serotonin (5-HT) signaling in multisensory function is poorly understood. To begin to fill this knowledge void, the current study provides physiological evidence for the influences of 5-HT signaling on auditory, visual and audiovisual responses of individual neurons in the intermediate and deep layers of the SC, with a focus on the 5HT2a receptor. Using single-unit extracellular recordings in combination with pharmacological methods, we demonstrate that alterations in 5HT2a receptor signaling change receptive field (RF) architecture as well as responsivity and integrative abilities of SC neurons when assessed at the level of the single neuron. In contrast, little changes were seen in the local field potential (LFP). These results are the first to implicate the serotonergic system in multisensory processing, and are an important step to understanding how modulatory networks mediate multisensory integration in the SC.

Intertrial Variability in the Premotor Cortex Accounts for Individual Differences in Peripersonal Space

We live in a dynamic environment, constantly confronted with approaching objects that we may either avoid or be forced to address. A multisensory and sensorimotor interface, the peripersonal space (PPS), mediates every physical interaction between our body and the environment. Behavioral investigations show high variability in the extension of PPS across individuals, but there is a lack of evidence on the neural underpinnings of these large individual differences. Here, we used approaching auditory stimuli and fMRI to capture the individual boundary of PPS and examine its neural underpinnings. Precisely, we tested the hypothesis that intertrial variability (ITV) in brain regions coding PPS predicts individual differences of its boundary at the behavioral level. Selectively in the premotor cortex, we found that ITV, rather than trial-averaged amplitude, of BOLD responses to far rather than near dynamic stimuli predicts the individual extension of PPS. Our results provide the first empirical support for the relevance of ITV of brain responses for individual differences in human behavior.

SIGNIFICANCE STATEMENT

Peripersonal space (PPS) is a multisensory and sensorimotor interface mediating every physical interaction between the body and the environment. A major characteristic of the boundary of PPS in humans is the extremely high variability of its location across individuals. We show that interindividual differences in the extension of the PPS are predicted by variability of BOLD responses in the premotor cortex to far stimuli approaching our body. Our results provide the first empirical support to the relevance of variability of evoked responses for human behavior and its variance across individuals.

Developmental time course of peripheral cross-modal sensory interaction of the trigeminal and gustatory systems

Few sensory modalities appear to engage in cross-modal interactions within the peripheral nervous system, making the integrated relationship between the peripheral gustatory and trigeminal systems an ideal model for investigating cross-sensory support. The present study examined taste system anatomy following unilateral transection of the trigeminal lingual nerve (LX) while leaving the gustatory chorda tympani intact. At 10, 25, or 65 days of age, rats underwent LX with outcomes assessed following various survival times. Fungiform papillae were classified by morphological feature using surface analysis. Taste bud volumes were calculated from histological sections of the anterior tongue. Differences in papillae morphology were evident by 2 days post-transection of P10 rats and by 8 days post in P25 rats. When transected at P65, animals never exhibited statistically significant morphological changes. After LX at P10, fewer taste buds were present on the transected side following 16 and 24 days survival time and remaining taste buds were smaller than on the intact side. In P25 and P65 animals, taste bud volumes were reduced on the denervated side by 8 and 16 days postsurgery, respectively. By 50 days post-transection, taste buds of P10 animals had not recovered in size; however, all observed changes in papillae morphology and taste buds subsided in P25 and P65 rats. Results indicate that LX impacts taste receptor cells and alters epithelial morphology of fungiform papillae, particularly during early development. These findings highlight dual roles for the lingual nerve in the maintenance of both gustatory and non-gustatory tissues on the anterior tongue.

Deficits in audiovisual speech perception in normal aging emerge at the level of whole-word recognition

Over the next 2 decades, a dramatic shift in the demographics of society will take place, with a rapid growth in the population of older adults. One of the most common complaints with healthy aging is a decreased ability to successfully perceive speech, particularly in noisy environments. In such noisy environments, the presence of visual speech cues (i.e., lip movements) provide striking benefits for speech perception and comprehension, but previous research suggests that older adults gain less from such audiovisual integration than their younger peers. To determine at what processing level these behavioral differences arise in healthy-aging populations, we administered a speech-in-noise task to younger and older adults. We compared the perceptual benefits of having speech information available in both the auditory and visual modalities and examined both phoneme and whole-word recognition across varying levels of signal-to-noise ratio. For whole-word recognition, older adults relative to younger adults showed greater multisensory gains at intermediate SNRs but reduced benefit at low SNRs. By contrast, at the phoneme level both younger and older adults showed approximately equivalent increases in multisensory gain as signal-to-noise ratio decreased. Collectively, the results provide important insights into both the similarities and differences in how older and younger adults integrate auditory and visual speech cues in noisy environments and help explain some of the conflicting findings in previous studies of multisensory speech perception in healthy aging. These novel findings suggest that audiovisual processing is intact at more elementary levels of speech perception in healthy-aging populations and that deficits begin to emerge only at the more complex word-recognition level of speech signals.

The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities

Behavior, perception and cognition are strongly shaped by the synthesis of information across the different sensory modalities. Such multisensory integration often results in performance and perceptual benefits that reflect the additional information conferred by having cues from multiple senses providing redundant or complementary information. The spatial and temporal relationships of these cues provide powerful statistical information about how these cues should be integrated or "bound" in order to create a unified perceptual representation. Much recent work has examined the temporal factors that are integral in multisensory processing, with many focused on the construct of the multisensory temporal binding window - the epoch of time within which stimuli from different modalities is likely to be integrated and perceptually bound. Emerging evidence suggests that this temporal window is altered in a series of neurodevelopmental disorders, including autism, dyslexia and schizophrenia. In addition to their role in sensory processing, these deficits in multisensory temporal function may play an important role in the perceptual and cognitive weaknesses that characterize these clinical disorders. Within this context, focus on improving the acuity of multisensory temporal function may have important implications for the amelioration of the "higher-order" deficits that serve as the defining features of these disorders.

Processing of sub- and supra-second intervals in the primate brain results from the calibration of neuronal oscillators via sensory, motor, and feedback processes

The processing of time intervals in the sub- to supra-second range by the brain is critical for the interaction of primates with their surroundings in activities, such as foraging and hunting. For an accurate processing of time intervals by the brain, representation of physical time within neuronal circuits is necessary. I propose that time dimension of the physical surrounding is represented in the brain by different types of neuronal oscillators, generating spikes or spike bursts at regular intervals. The proposed oscillators include the pacemaker neurons, tonic inputs, and synchronized excitation and inhibition of inter-connected neurons. Oscillators, which are built inside various circuits of brain, help to form modular clocks, processing time intervals or other temporal characteristics specific to functions of a circuit. Relative or absolute duration is represented within neuronal oscillators by "neural temporal unit," defined as the interval between regularly occurring spikes or spike bursts. Oscillator output is processed to produce changes in activities of neurons, named frequency modulator neuron, wired within a separate module, represented by the rate of change in frequency, and frequency of activities, proposed to encode time intervals. Inbuilt oscillators are calibrated by (a) feedback processes, (b) input of time intervals resulting from rhythmic external sensory stimulation, and (c) synchronous effects of feedback processes and evoked sensory activity. A single active clock is proposed per circuit, which is calibrated by one or more mechanisms. Multiple calibration mechanisms, inbuilt oscillators, and the presence of modular connections prevent a complete loss of interval timing functions of the brain.