Audio-visual experience strengthens multisensory assemblies in adult mouse visual cortex

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active research area. Inspired by the cortico-thalamo-cortical circuit, in which the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus, we propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS). First, the CTCNet learns hierarchical auditory and visual representations in a bottom-up manner in separate auditory and visual subnetworks, mimicking the functions of the auditory and visual cortical areas. Then, inspired by the large number of connections between cortical regions and the thalamus, the model fuses the auditory and visual information in a thalamic subnetwork through top-down connections. Finally, the model transmits this fused information back to the auditory and visual subnetworks, and the above process is repeated several times. The results of experiments on three speech separation benchmark datasets show that CTCNet remarkably outperforms existing AVSS methods with considerably fewer parameters. These results suggest that mimicking the anatomical connectome of the mammalian brain has great potential for advancing the development of deep neural networks.

Network state changes in sensory thalamus represent learned outcomes

Thalamic brain areas play an important role in adaptive behaviors. Nevertheless, the population dynamics of thalamic relays during learning across sensory modalities remain unknown. Using a cross-modal sensory reward-associative learning paradigm combined with deep brain two-photon calcium imaging of large populations of auditory thalamus (medial geniculate body, MGB) neurons in male mice, we identified that MGB neurons are biased towards reward predictors independent of modality. Additionally, functional classes of MGB neurons aligned with distinct task periods and behavioral outcomes, both dependent and independent of sensory modality. During non-sensory delay periods, MGB ensembles developed coherent neuronal representation as well as distinct co-activity network states reflecting predicted task outcome. These results demonstrate flexible cross-modal ensemble coding in auditory thalamus during adaptive learning and highlight its importance in brain-wide cross-modal computations during complex behavior.

Stability of cross-sensory input to primary somatosensory cortex across experience

Merging information from across sensory modalities is key to forming robust, disambiguated percepts of the world, yet how the brain achieves this feat remains unclear. Recent observations of cross-modal influences in primary sensory cortical areas have suggested that multisensory integration may occur in the earliest stages of cortical processing, but the role of these responses is still poorly understood. We address these questions by testing several hypotheses about the possible functions served by auditory influences on the barrel field of mouse primary somatosensory cortex (S1) using in vivo 2-photon calcium imaging. We observed sound-evoked spiking activity in a small fraction of cells overall, and moreover that this sparse activity was insufficient to encode auditory stimulus identity; few cells responded preferentially to one sound or another, and a linear classifier trained to decode auditory stimuli from population activity performed barely above chance. Moreover S1 did not encode information about specific audio-tactile feature conjunctions that we tested. Our ability to decode auditory audio-tactile stimuli from neural activity remained unchanged after both passive experience and reinforcement. Collectively, these results suggest that while a primary sensory cortex is highly plastic with regard to its own modality, the influence of other modalities are remarkably stable and play a largely stimulus-non-specific role.

Auditory cortex conveys non-topographic sound localization signals to visual cortex

Spatiotemporally congruent sensory stimuli are fused into a unified percept. The auditory cortex (AC) sends projections to the primary visual cortex (V1), which could provide signals for binding spatially corresponding audio-visual stimuli. However, whether AC inputs in V1 encode sound location remains unknown. Using two-photon axonal calcium imaging and a speaker array, we measured the auditory spatial information transmitted from AC to layer 1 of V1. AC conveys information about the location of ipsilateral and contralateral sound sources to V1. Sound location could be accurately decoded by sampling AC axons in V1, providing a substrate for making location-specific audiovisual associations. However, AC inputs were not retinotopically arranged in V1, and audio-visual modulations of V1 neurons did not depend on the spatial congruency of the sound and light stimuli. The non-topographic sound localization signals provided by AC might allow the association of specific audiovisual spatial patterns in V1 neurons.

Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex

Primary sensory cortices respond to crossmodal stimuli-for example, auditory responses are found in primary visual cortex (V1). However, it remains unclear whether these responses reflect sensory inputs or behavioral modulation through sound-evoked body movement. We address this controversy by showing that sound-evoked activity in V1 of awake mice can be dissociated into auditory and behavioral components with distinct spatiotemporal profiles. The auditory component began at approximately 27 ms, was found in superficial and deep layers and originated from auditory cortex. Sound-evoked orofacial movements correlated with V1 neural activity starting at approximately 80-100 ms and explained auditory frequency tuning. Visual, auditory and motor activity were expressed by different laminar profiles and largely segregated subsets of neuronal populations. During simultaneous audiovisual stimulation, visual representations remained dissociable from auditory-related and motor-related activity. This three-fold dissociability of auditory, motor and visual processing is central to understanding how distinct inputs to visual cortex interact to support vision.

Sound suppresses earliest visual cortical processing after sight recovery in congenitally blind humans

Neuroscientific research has consistently shown more extensive non-visual activity in the visual cortex of congenitally blind humans compared to sighted controls; a phenomenon known as crossmodal plasticity. Whether or not crossmodal activation of the visual cortex retracts if sight can be restored is still unknown. The present study, involving a rare group of sight-recovery individuals who were born pattern vision blind, employed visual event-related potentials to investigate persisting crossmodal modulation of the initial visual cortical processing stages. Here we report that the earliest, stimulus-driven retinotopic visual cortical activity (<100 ms) was suppressed in a spatially specific manner in sight-recovery individuals when concomitant sounds accompanied visual stimulation. In contrast, sounds did not modulate the earliest visual cortical response in two groups of typically sighted controls, nor in a third control group of sight-recovery individuals who had suffered a transient phase of later (rather than congenital) visual impairment. These results provide strong evidence for persisting crossmodal activity in the visual cortex after sight recovery following a period of congenital visual deprivation. Based on the time course of this modulation, we speculate on a role of exuberant crossmodal thalamic input which may arise during a sensitive phase of brain development.

The Development of Multisensory Integration at the Neuronal Level

Multisensory integration is a fundamental function of the brain. In the typical adult, multisensory neurons' response to paired multisensory (e.g., audiovisual) cues is significantly more robust than the corresponding best unisensory response in many brain regions. Synthesizing sensory signals from multiple modalities can speed up sensory processing and improve the salience of outside events or objects. Despite its significance, multisensory integration is testified to be not a neonatal feature of the brain. Neurons' ability to effectively combine multisensory information does not occur rapidly but develops gradually during early postnatal life (for cats, 4-12 weeks required). Multisensory experience is critical for this developing process. If animals were restricted from sensing normal visual scenes or sounds (deprived of the relevant multisensory experience), the development of the corresponding integrative ability could be blocked until the appropriate multisensory experience is obtained. This section summarizes the extant literature on the development of multisensory integration (mainly using cat superior colliculus as a model), sensory-deprivation-induced cross-modal plasticity, and how sensory experience (sensory exposure and perceptual learning) leads to the plastic change and modification of neural circuits in cortical and subcortical areas.

Age-related dysregulation of homeostatic control in neuronal microcircuits

Neuronal homeostasis prevents hyperactivity and hypoactivity. Age-related hyperactivity suggests homeostasis may be dysregulated in later life. However, plasticity mechanisms preventing age-related hyperactivity and their efficacy in later life are unclear. We identify the adult cortical plasticity response to elevated activity driven by sensory overstimulation, then test how plasticity changes with age. We use in vivo two-photon imaging of calcium-mediated cellular/synaptic activity, electrophysiology and c-Fos-activity tagging to show control of neuronal activity is dysregulated in the visual cortex in late adulthood. Specifically, in young adult cortex, mGluR5-dependent population-wide excitatory synaptic weakening and inhibitory synaptogenesis reduce cortical activity following overstimulation. In later life, these mechanisms are downregulated, so that overstimulation results in synaptic strengthening and elevated activity. We also find overstimulation disrupts cognition in older but not younger animals. We propose that specific plasticity mechanisms fail in later life dysregulating neuronal microcircuit homeostasis and that the age-related response to overstimulation can impact cognitive performance.

Widefield imaging of rapid pan-cortical voltage dynamics with an indicator evolved for one-photon microscopy

Widefield imaging with genetically encoded voltage indicators (GEVIs) is a promising approach for understanding the role of large cortical networks in the neural coding of behavior. However, the limited performance of current GEVIs restricts their deployment for single-trial imaging of rapid neuronal voltage dynamics. Here, we developed a high-throughput platform to screen for GEVIs that combine fast kinetics with high brightness, sensitivity, and photostability under widefield one-photon illumination. Rounds of directed evolution produced JEDI-1P, a green-emitting fluorescent indicator with enhanced performance across all metrics. Next, we optimized a neonatal intracerebroventricular delivery method to achieve cost-effective and wide-spread JEDI-1P expression in mice. We also developed an approach to correct optical measurements from hemodynamic and motion artifacts effectively. Finally, we achieved stable brain-wide voltage imaging and successfully tracked gamma-frequency whisker and visual stimulations in awake mice in single trials, opening the door to investigating the role of high-frequency signals in brain computations.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

How 'visual' is the visual cortex? The interactions between the visual cortex and other sensory, motivational and motor systems as enabling factors for visual perception

The definition of the visual cortex is primarily based on the evidence that lesions of this area impair visual perception. However, this does not exclude that the visual cortex may process more information than of retinal origin alone, or that other brain structures contribute to vision. Indeed, research across the past decades has shown that non-visual information, such as neural activity related to reward expectation and value, locomotion, working memory and other sensory modalities, can modulate primary visual cortical responses to retinal inputs. Nevertheless, the function of this non-visual information is poorly understood. Here we review recent evidence, coming primarily from studies in rodents, arguing that non-visual and motor effects in visual cortex play a role in visual processing itself, for instance disentangling direct auditory effects on visual cortex from effects of sound-evoked orofacial movement. These findings are placed in a broader framework casting vision in terms of predictive processing under control of frontal, reward- and motor-related systems. In contrast to the prevalent notion that vision is exclusively constructed by the visual cortical system, we propose that visual percepts are generated by a larger network-the extended visual system-spanning other sensory cortices, supramodal areas and frontal systems. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Coherent mapping of position and head direction across auditory and visual cortex

Neurons in primary visual cortex (V1) may not only signal current visual input but also relevant contextual information such as reward expectancy and the subject's spatial position. Such contextual representations need not be restricted to V1 but could participate in a coherent mapping throughout sensory cortices. Here, we show that spiking activity coherently represents a location-specific mapping across auditory cortex (AC) and lateral, secondary visual cortex (V2L) of freely moving rats engaged in a sensory detection task on a figure-8 maze. Single-unit activity of both areas showed extensive similarities in terms of spatial distribution, reliability, and position coding. Importantly, reconstructions of subject position based on spiking activity displayed decoding errors that were correlated between areas. Additionally, we found that head direction, but not locomotor speed or head angular velocity, was an important determinant of activity in AC and V2L. By contrast, variables related to the sensory task cues or to trial correctness and reward were not markedly encoded in AC and V2L. We conclude that sensory cortices participate in coherent, multimodal representations of the subject's sensory-specific location. These may provide a common reference frame for distributed cortical sensory and motor processes and may support crossmodal predictive processing.

Single neurons in the human medial temporal lobe flexibly shift representations across spatial and memory tasks

Investigations into how individual neurons encode behavioral variables of interest have revealed specific representations in single neurons, such as place and object cells, as well as a wide range of cells with conjunctive encodings or mixed selectivity. However, as most experiments examine neural activity within individual tasks, it is currently unclear if and how neural representations change across different task contexts. Within this discussion, the medial temporal lobe is particularly salient, as it is known to be important for multiple behaviors including spatial navigation and memory, however the relationship between these functions is currently unclear. Here, to investigate how representations in single neurons vary across different task contexts in the medial temporal lobe, we collected and analyzed single-neuron activity from human participants as they completed a paired-task session consisting of a passive-viewing visual working memory and a spatial navigation and memory task. Five patients contributed 22 paired-task sessions, which were spike sorted together to allow for the same putative single neurons to be compared between the different tasks. Within each task, we replicated concept-related activations in the working memory task, as well as target-location and serial-position responsive cells in the navigation task. When comparing neuronal activity between tasks, we first established that a significant number of neurons maintained the same kind of representation, responding to stimuli presentations across tasks. Further, we found cells that changed the nature of their representation across tasks, including a significant number of cells that were stimulus responsive in the working memory task that responded to serial position in the spatial task. Overall, our results support a flexible encoding of multiple, distinct aspects of different tasks by single neurons in the human medial temporal lobe, whereby some individual neurons change the nature of their feature coding between task contexts.

Multisensory learning binds neurons into a cross-modal memory engram

Associating multiple sensory cues with objects and experience is a fundamental brain process that improves object recognition and memory performance. However, neural mechanisms that bind sensory features during learning and augment memory expression are unknown. Here we demonstrate multisensory appetitive and aversive memory in Drosophila. Combining colours and odours improved memory performance, even when each sensory modality was tested alone. Temporal control of neuronal function revealed visually selective mushroom body Kenyon cells (KCs) to be required for enhancement of both visual and olfactory memory after multisensory training. Voltage imaging in head-fixed flies showed that multisensory learning binds activity between streams of modality-specific KCs so that unimodal sensory input generates a multimodal neuronal response. Binding occurs between regions of the olfactory and visual KC axons, which receive valence-relevant dopaminergic reinforcement, and is propagated downstream. Dopamine locally releases GABAergic inhibition to permit specific microcircuits within KC-spanning serotonergic neurons to function as an excitatory bridge between the previously 'modality-selective' KC streams. Cross-modal binding thereby expands the KCs representing the memory engram for each modality into those representing the other. This broadening of the engram improves memory performance after multisensory learning and permits a single sensory feature to retrieve the memory of the multimodal experience.

Sound Improves Neuronal Encoding of Visual Stimuli in Mouse Primary Visual Cortex

In everyday life, we integrate visual and auditory information in routine tasks such as navigation and communication. While concurrent sound can improve visual perception, the neuronal correlates of audiovisual integration are not fully understood. Specifically, it remains unclear whether neuronal firing patters in the primary visual cortex (V1) of awake animals demonstrate similar sound-induced improvement in visual discriminability. Furthermore, presentation of sound is associated with movement in the subjects, but little is understood about whether and how sound-associated movement affects audiovisual integration in V1. Here, we investigated how sound and movement interact to modulate V1 visual responses in awake, head-fixed mice and whether this interaction improves neuronal encoding of the visual stimulus. We presented visual drifting gratings with and without simultaneous auditory white noise to awake mice while recording mouse movement and V1 neuronal activity. Sound modulated activity of 80% of light-responsive neurons, with 95% of neurons increasing activity when the auditory stimulus was present. A generalized linear model (GLM) revealed that sound and movement had distinct and complementary effects of the neuronal visual responses. Furthermore, decoding of the visual stimulus from the neuronal activity was improved with sound, an effect that persisted even when controlling for movement. These results demonstrate that sound and movement modulate visual responses in complementary ways, improving neuronal representation of the visual stimulus. This study clarifies the role of movement as a potential confound in neuronal audiovisual responses and expands our knowledge of how multimodal processing is mediated at a neuronal level in the awake brain.SIGNIFICANCE STATEMENT Sound and movement are both known to modulate visual responses in the primary visual cortex; however, sound-induced movement has largely remained unaccounted for as a potential confound in audiovisual studies in awake animals. Here, authors found that sound and movement both modulate visual responses in an important visual brain area, the primary visual cortex, in distinct, yet complementary ways. Furthermore, sound improved encoding of the visual stimulus even when accounting for movement. This study reconciles contrasting theories on the mechanism underlying audiovisual integration and asserts the primary visual cortex as a key brain region participating in tripartite sensory interactions.

Age-related changes in excitatory and inhibitory intra-cortical circuits in auditory cortex of C57Bl/6 mice

A common impairment in aging is age-related hearing loss (presbycusis), which manifests as impaired spectrotemporal processing. Aging is accompanied by alteration in normal inhibitory (GABA) neurotransmission, and changes in excitatory (NMDA and AMPA) synapses in the auditory cortex (ACtx). However, the circuits affected by these synaptic changes remain unknown. Mice of the C57Bl/6J strain show premature age-related hearing loss and changes in functional responses in ACtx. We thus investigated how auditory cortical microcircuits change with age by comparing young (∼ 6 weeks) and aged (>1 year old) C57Bl/6J mice. We performed laser scanning photostimulation (LSPS) combined with whole-cell patch clamp recordings from Layer (L) 2/3 cells in primary auditory cortex (A1) of young adult and aged C57Bl/6J mice. We found that L2/3 cells in aged C57Bl/6J mice display functional hypoconnectivity of both excitatory and inhibitory circuits. Compared to cells from young C57Bl/6 mice, cells from aged C57Bl/6J mice have fewer excitatory connections with weaker connection strength. Whereas young adult and aged C57Bl/6J mice have similar amounts of inhibitory connections, the strength of local inhibition is weaker in the aged group. We confirmed these results by recording miniature excitatory (mEPSCs) and inhibitory synaptic currents (mIPSCs). Our results suggest a specific reduction in excitatory and inhibitory intralaminar cortical circuits in aged C57Bl/6J mice compared with young adult animals. We speculate that these unbalanced changes in cortical circuits contribute to the functional manifestations of age-related hearing loss.

Behavioral origin of sound-evoked activity in mouse visual cortex

Sensory cortices can be affected by stimuli of multiple modalities and are thus increasingly thought to be multisensory. For instance, primary visual cortex (V1) is influenced not only by images but also by sounds. Here we show that the activity evoked by sounds in V1, measured with Neuropixels probes, is stereotyped across neurons and even across mice. It is independent of projections from auditory cortex and resembles activity evoked in the hippocampal formation, which receives little direct auditory input. Its low-dimensional nature starkly contrasts the high-dimensional code that V1 uses to represent images. Furthermore, this sound-evoked activity can be precisely predicted by small body movements that are elicited by each sound and are stereotyped across trials and mice. Thus, neural activity that is apparently multisensory may simply arise from low-dimensional signals associated with internal state and behavior.

Age-related decrease in motor contribution to multisensory reaction times in primary school children

Traditional measurement of multisensory facilitation in tasks such as speeded motor reaction tasks (MRT) consistently show age-related improvement during early childhood. However, the extent to which motor function increases with age and hence contribute to multisensory motor reaction times in young children has seldom been examined. Thus, we aimed to investigate the contribution of motor development to measures of multisensory (auditory, visual, and audiovisual) and visuomotor processing tasks in three young school age groups of children (n = 69) aged (5-6, n = 21; 7-8, n = 25.; 9-10 n = 18 years). We also aimed to determine whether age-related sensory threshold times for purely visual inspection time (IT) tasks improved significantly with age. Bayesian results showed decisive evidence for age-group differences in multisensory MRT and visuo-motor processing tasks, though the evidence showed that threshold time for visual identification IT performance was only slower in the youngest age group children (5-6) compared to older groups. Bayesian correlations between performance on the multisensory MRT and visuo-motor processing tasks indicated moderate to decisive evidence in favor of the alternative hypothesis (BF10 = 4.71 to 91.346), though not with the threshold IT (BF10 < 1.35). This suggests that visual sensory system development in children older than 6 years makes a less significant contribution to the measure of multisensory facilitation, compared to motor development. In addition to this main finding, multisensory facilitation of MRT within race-model predictions was only found in the oldest group of children (9-10), supporting previous suggestions that multisensory integration is likely to continue into late childhood/early adolescence at least.

Cupid's quiver: Integrating sensory cues in rodent mating systems

In many animal species, males and females exploit different mating strategies, display sex-typical behaviors, and use distinct systems to recognize ethologically relevant cues. Mate selection thus requires mutual recognition across diverse social interactions based on distinct sensory signals. These sex differences in courtship and mating behaviors correspond to differences in sensory systems and downstream neural substrates engaged to recognize and respond to courtship signals. In many rodents, males tend to rely heavily on volatile olfactory and pheromone cues, while females appear to be guided more by a combination of these chemosensory signals with acoustic cues in the form of ultrasonic vocalizations. The mechanisms by which chemical and acoustic cues are integrated to control behavior are understudied in mating but are known to be important in the control of maternal behaviors. Socially monogamous species constitute a behaviorally distinct group of rodents. In these species, anatomic differences between males and females outside the nervous system are less prominent than in species with non-monogamous mating systems, and both sexes engage in more symmetric social behaviors and form attachments. Nevertheless, despite the apparent similarities in behaviors displayed by monogamous males and females, the circuitry supporting social, mating, and attachment behaviors in these species is increasingly thought to differ between the sexes. Sex differences in sensory modalities most important for mate recognition in across species are of particular interest and present a wealth of questions yet to be answered. Here, we discuss how distinct sensory cues may be integrated to drive social and attachment behaviors in rodents, and the differing roles of specific sensory systems in eliciting displays of behavior by females or males.

Low frequency visual stimulation enhances slow wave activity without disrupting the sleep pattern in mice

Non-invasive stimulation technologies are emerging as potential treatment options for a range of neurodegenerative disorders. Experimental evidence suggests that stimuli-evoked changes in slow brain rhythms may mitigate or even prevent neuropathological and behavioral impairments. Slow wave activity is prevalent during sleep and can be triggered non-invasively by sensory stimulation targeting the visual system or directly via activation of neurons locally using optogenetics. Here, we developed new tools for delivering visual stimulation using light-emitting diodes in freely moving mice while awake and during sleep. We compared these tools to traditional optogenetic approaches used for local stimulation of neurons in the cerebral cortex. We then used these tools to compare the effects of low-frequency visual versus optogenetic stimulations on the slow wave activity and sleep pattern in mice. Visual stimulation effectively enhanced slow wave activity without disrupting the sleep pattern. Optogenetic stimulation of cortical GABAergic neurons increased NREM sleep. These results suggest that visual stimulation can be effective at boosting slow wave activity without having adverse effects on sleep and thus holds great potential as a non-invasive stimulation treatment strategy.

Existing function in primary visual cortex is not perturbed by new skill acquisition of a non-matched sensory task

Acquisition of new skills has the potential to disturb existing network function. To directly assess whether previously acquired cortical function is altered during learning, mice were trained in an abstract task in which selected activity patterns were rewarded using an optical brain-computer interface device coupled to primary visual cortex (V1) neurons. Excitatory neurons were longitudinally recorded using 2-photon calcium imaging. Despite significant changes in local neural activity during task performance, tuning properties and stimulus encoding assessed outside of the trained context were not perturbed. Similarly, stimulus tuning was stable in neurons that remained responsive following a different, visual discrimination training task. However, visual discrimination training increased the rate of representational drift. Our results indicate that while some forms of perceptual learning may modify the contribution of individual neurons to stimulus encoding, new skill learning is not inherently disruptive to the quality of stimulus representation in adult V1.

Estimating How Sounds Modulate Orientation Representation in the Primary Visual Cortex Using Shallow Neural Networks

Audiovisual perception results from the interaction between visual and auditory processing. Hence, presenting auditory and visual inputs simultaneously usually improves the accuracy of the unimodal percepts, but can also lead to audiovisual illusions. Cross-talks between visual and auditory inputs during sensory processing were recently shown to occur as early as in the primary visual cortex (V1). In a previous study, we demonstrated that sounds improve the representation of the orientation of visual stimuli in the naïve mouse V1 by promoting the recruitment of neurons better tuned to the orientation and direction of the visual stimulus. However, we did not test if this type of modulation was still present when the auditory and visual stimuli were both behaviorally relevant. To determine the effect of sounds on active visual processing, we performed calcium imaging in V1 while mice were performing an audiovisual task. We then compared the representations of the task stimuli orientations in the unimodal visual and audiovisual context using shallow neural networks (SNNs). SNNs were chosen because of the biological plausibility of their computational structure and the possibility of identifying post hoc the biological neurons having the strongest influence on the classification decision. We first showed that SNNs can categorize the activity of V1 neurons evoked by drifting gratings of 12 different orientations. Then, we demonstrated using the connection weight approach that SNN training assigns the largest computational weight to the V1 neurons having the best orientation and direction selectivity. Finally, we showed that it is possible to use SNNs to determine how V1 neurons represent the orientations of stimuli that do not belong to the set of orientations used for SNN training. Once the SNN approach was established, we replicated the previous finding that sounds improve orientation representation in the V1 of naïve mice. Then, we showed that, in mice performing an audiovisual detection task, task tones improve the representation of the visual cues associated with the reward while deteriorating the representation of non-rewarded cues. Altogether, our results suggest that the direction of sound modulation in V1 depends on the behavioral relevance of the visual cue.

Deficits in Sense of Body Ownership, Sensory Processing, and Temporal Perception in Schizophrenia Patients With/Without Auditory Verbal Hallucinations

It has been claimed that individuals with schizophrenia have difficulty in self-recognition and, consequently, are unable to identify the sources of their sensory perceptions or thoughts, resulting in delusions, hallucinations, and unusual experiences of body ownership. The deficits also contribute to the enhanced rubber hand illusion (RHI; a body perception illusion, induced by synchronous visual and tactile stimulation). Evidence based on RHI paradigms is emerging that auditory information can make an impact on the sense of body ownership, which relies on the process of multisensory inputs and integration. Hence, we assumed that auditory verbal hallucinations (AVHs), as an abnormal auditory perception, could be linked with body ownership, and the RHI paradigm could be conducted in patients with AVHs to explore the underlying mechanisms. In this study, we investigated the performance of patients with/without AVHs in the RHI. We administered the RHI paradigm to 80 patients with schizophrenia (47 with AVHs and 33 without AVHs) and 36 healthy controls. We conducted the experiment under two conditions (synchronous and asynchronous) and evaluated the RHI effects by both objective and subjective measures. Both patient groups experienced the RHI more quickly and strongly than HCs. The RHI effects of patients with AVHs were significantly smaller than those of patients without AVHs. Another important finding was that patients with AVHs did not show a reduction in RHI under asynchronous conditions. These results emphasize the disturbances of the sense of body ownership in schizophrenia patients with/without AVHs and the associations with AVHs. Furthermore, it is suggested that patients with AVHs may have multisensory processing dysfunctions and internal timing deficits.

Homeostatic regulation through strengthening of neuronal network-correlated synaptic inputs

Homeostatic regulation is essential for stable neuronal function. Several synaptic mechanisms of homeostatic plasticity have been described, but the functional properties of synapses involved in homeostasis are unknown. We used longitudinal two-photon functional imaging of dendritic spine calcium signals in visual and retrosplenial cortices of awake adult mice to quantify the sensory deprivation-induced changes in the responses of functionally identified spines. We found that spines whose activity selectively correlated with intrinsic network activity underwent tumor necrosis factor alpha (TNF-α)-dependent homeostatic increases in their response amplitudes, but spines identified as responsive to sensory stimulation did not. We observed an increase in the global sensory-evoked responses following sensory deprivation, despite the fact that the identified sensory inputs did not strengthen. Instead, global sensory-evoked responses correlated with the strength of network-correlated inputs. Our results suggest that homeostatic regulation of global responses is mediated through changes to intrinsic network-correlated inputs rather than changes to identified sensory inputs thought to drive sensory processing.

Cholinergic modulation of sensory processing in awake mouse cortex

Cholinergic modulation of brain activity is fundamental for awareness and conscious sensorimotor behaviours, but deciphering the timing and significance of acetylcholine actions for these behaviours is challenging. The widespread nature of cholinergic projections to the cortex means that new insights require access to specific neuronal populations, and on a time-scale that matches behaviourally relevant cholinergic actions. Here, we use fast, voltage imaging of L2/3 cortical pyramidal neurons exclusively expressing the genetically-encoded voltage indicator Butterfly 1.2, in awake, head-fixed mice, receiving sensory stimulation, whilst manipulating the cholinergic system. Altering muscarinic acetylcholine function re-shaped sensory-evoked fast depolarisation and subsequent slow hyperpolarisation of L2/3 pyramidal neurons. A consequence of this re-shaping was disrupted adaptation of the sensory-evoked responses, suggesting a critical role for acetylcholine during sensory discrimination behaviour. Our findings provide new insights into how the cortex processes sensory information and how loss of acetylcholine, for example in Alzheimer's Disease, disrupts sensory behaviours.

Multisensory-Guided Associative Learning Enhances Multisensory Representation in Primary Auditory Cortex

Sensory cortices, classically considered to represent modality-specific sensory information, are also found to engage in multisensory processing. However, how sensory processing in sensory cortices is cross-modally modulated remains an open question. Specifically, we understand little of cross-modal representation in sensory cortices in perceptual tasks and how perceptual learning modifies this process. Here, we recorded neural responses in primary auditory cortex (A1) both while freely moving rats discriminated stimuli in Go/No-Go tasks and when anesthetized. Our data show that cross-modal representation in auditory cortices varies with task contexts. In the task of an audiovisual cue being the target associating with water reward, a significantly higher proportion of auditory neurons showed a visually evoked response. The vast majority of auditory neurons, if processing auditory-visual interactions, exhibit significant multisensory enhancement. However, when the rats performed tasks with unisensory cues being the target, cross-modal inhibition, rather than enhancement, predominated. In addition, multisensory associational learning appeared to leave a trace of plastic change in A1, as a larger proportion of A1 neurons showed multisensory enhancement in anesthesia. These findings indicate that multisensory processing in principle sensory cortices is not static, and having cross-modal interaction in the task requirement can substantially enhance multisensory processing in sensory cortices.

Coming full circle: In vivo Veritas, or expanding the neuroscience frontier

The 25-century long history of brain science can be divided into four distinct 'Eras': Era 1 (∼2000 years) started around 500 BC with reductionist studies of human and animal brains using invasive in vivo and ex vivo methods; Era 2 (∼200 years) started in the 17^th century and introduced the first invasive methods to study the function of living tissues; Era 3 started around 1838 and is the ongoing era of cellular neurophysiology; finally, Era 4 (in statu nascendi) is the era of non-invasive, holistic yet mechanistic, studies of the brain. Animal experimentation is becoming increasingly more holistic as multimodal imaging and recording techniques are combined in a single experiment on the brain of awake behaving animal. This newly emerged approach can be called in vigilo (from Latin 'in awake' or 'in vigilant state'), by analogy to the earlier introduced terms in vivo or in vitro. We introduce the Special Issue "In Vigilo Veritas: New Frontiers of Optical Imaging and Electrical Recording in the Brain of Awake Behaving Mice", which features original research articles and reviews that represent some of the finest examples of a truly multimodal studies, where behavioural readouts and tasks are combined in the same longitudinal experiment and on the same mouse with the two-photon imaging, optogenetics and/or electrophysiological recordings. This exciting multi-methodological approach creates a fertile ground for breakthrough discoveries in neurophysiology and neuropsychology of an awake behaving mammalian brain.

Bioinspired multisensory neural network with crossmodal integration and recognition

The integration and interaction of vision, touch, hearing, smell, and taste in the human multisensory neural network facilitate high-level cognitive functionalities, such as crossmodal integration, recognition, and imagination for accurate evaluation and comprehensive understanding of the multimodal world. Here, we report a bioinspired multisensory neural network that integrates artificial optic, afferent, auditory, and simulated olfactory and gustatory sensory nerves. With distributed multiple sensors and biomimetic hierarchical architectures, our system can not only sense, process, and memorize multimodal information, but also fuse multisensory data at hardware and software level. Using crossmodal learning, the system is capable of crossmodally recognizing and imagining multimodal information, such as visualizing alphabet letters upon handwritten input, recognizing multimodal visual/smell/taste information or imagining a never-seen picture when hearing its description. Our multisensory neural network provides a promising approach towards robotic sensing and perception.

Multi-scale network imaging in a mouse model of amyloidosis

The adult neocortex is not hard-wired but instead retains the capacity to reorganise across multiple spatial scales long into adulthood. Plastic reorganisation occurs at the level of mesoscopic sensory maps, functional neuronal assemblies and synaptic ensembles and is thought to be a critical feature of neuronal network function. Here, we describe a series of approaches that use calcium imaging to measure network reorganisation across multiple spatial scales in vivo. At the mesoscopic level, we demonstrate that sensory activity can be measured in animals undergoing longitudinal behavioural assessment involving automated touchscreen tasks. At the cellular level, we show that network dynamics can be longitudinally measured at both stable and transient functional assemblies. At the level of single synapses, we show that functional subcellular calcium imaging approaches can be used to measure synaptic ensembles of dendritic spines in vivo. Finally, we demonstrate that all three levels of imaging can be spatially related to local pathology in a preclinical rodent model of amyloidosis. We propose that multi-scale in vivo calcium imaging can be used to measure parallel plasticity processes operating across multiple spatial scales in both the healthy brain and preclinical models of disease.

The aging mouse brain: cognition, connectivity and calcium

Aging is a complex process that differentially impacts multiple cognitive, sensory, neuronal and molecular processes. Technological innovations now allow for parallel investigation of neuronal circuit function, structure and molecular composition in the brain of awake behaving adult mice. Thus, mice have become a critical tool to better understand how aging impacts the brain. However, a more granular systems-based approach, which considers the impact of age on key features relating to neural processing, is required. Here, we review evidence probing the impact of age on the mouse brain. We focus on a range of processes relating to neuronal function, including cognitive abilities, sensory systems, synaptic plasticity and calcium regulation. Across many systems, we find evidence for prominent age-related dysregulation even before 12 months of age, suggesting that emerging age-related alterations can manifest by late adulthood. However, we also find reports suggesting that some processes are remarkably resilient to aging. The evidence suggests that aging does not drive a parallel, linear dysregulation of all systems, but instead impacts some processes earlier, and more severely, than others. We propose that capturing the more fine-scale emerging features of age-related vulnerability and resilience may provide better opportunities for the rejuvenation of the aged brain.

Approaches to Understanding Multisensory Dysfunction in Autism Spectrum Disorder

Abnormal sensory responses are a DSM-5 symptom of autism spectrum disorder (ASD), and research findings demonstrate altered sensory processing in ASD. Beyond difficulties with processing information within single sensory domains, including both hypersensitivity and hyposensitivity, difficulties in multisensory processing are becoming a core issue of focus in ASD. These difficulties may be targeted by treatment approaches such as "sensory integration," which is frequently applied in autism treatment but not yet based on clear evidence. Recently, psychophysical data have emerged to demonstrate multisensory deficits in some children with ASD. Unlike deficits in social communication, which are best understood in humans, sensory and multisensory changes offer a tractable marker of circuit dysfunction that is more easily translated into animal model systems to probe the underlying neurobiological mechanisms. Paralleling experimental paradigms that were previously applied in humans and larger mammals, we and others have demonstrated that multisensory function can also be examined behaviorally in rodents. Here, we review the sensory and multisensory difficulties commonly found in ASD, examining laboratory findings that relate these findings across species. Next, we discuss the known neurobiology of multisensory integration, drawing largely on experimental work in larger mammals, and extensions of these paradigms into rodents. Finally, we describe emerging investigations into multisensory processing in genetic mouse models related to autism risk. By detailing findings from humans to mice, we highlight the advantage of multisensory paradigms that can be easily translated across species, as well as the potential for rodent experimental systems to reveal opportunities for novel treatments. LAY SUMMARY: Sensory and multisensory deficits are commonly found in ASD and may result in cascading effects that impact social communication. By using similar experiments to those in humans, we discuss how studies in animal models may allow an understanding of the brain mechanisms that underlie difficulties in multisensory integration, with the ultimate goal of developing new treatments. Autism Res 2020, 13: 1430-1449. © 2020 International Society for Autism Research, Wiley Periodicals, Inc.