A cortical circuit for audio-visual predictions

Visual hallucinations originating in the retinofugal pathway under clinical and psychedelic conditions

Psychedelics like LSD (Lysergic acid diethylamide) and psilocybin are known to modulate perceptual modalities due to the activation of mostly serotonin receptors in specific cortical (e.g., visual cortex) and subcortical (e.g., thalamus) regions of the brain. In the visual domain, these psychedelic modulations often result in peculiar disturbances of viewed objects and light and sometimes even in hallucinations of non-existent environments, objects, and creatures. Although the underlying processes are poorly understood, research conducted over the past twenty years on the subjective experience of psychedelics details theories that attempt to explain these perceptual alterations due to a disruption of communication between cortical and subcortical regions. However, rare medical conditions in the visual system like Charles Bonnet syndrome that cause perceptual distortions may shed new light on the additional importance of the retinofugal pathway in psychedelic subjective experiences. Interneurons in the retina called amacrine cells could be the first site of visual psychedelic modulation and aid in disrupting the hierarchical structure of how humans perceive visual information. This paper presents an understanding of how the retinofugal pathway communicates and modulates visual information in psychedelic and clinical conditions. Therefore, we elucidate a new theory of psychedelic modulation in the retinofugal pathway.

Predictive Processing: A Circuit Approach to Psychosis

Predictive processing is a computational framework that aims to explain how the brain processes sensory information by making predictions about the environment and minimizing prediction errors. It can also be used to explain some of the key symptoms of psychotic disorders such as schizophrenia. In recent years, substantial advances have been made in our understanding of the neuronal circuitry that underlies predictive processing in cortex. In this review, we summarize these findings and how they might relate to psychosis and to observed cell type-specific effects of antipsychotic drugs. We argue that quantifying the effects of antipsychotic drugs on specific neuronal circuit elements is a promising approach to understanding not only the mechanism of action of antipsychotic drugs but also psychosis. Finally, we outline some of the key experiments that should be done. The aims of this review are to provide an overview of the current circuit-based approaches to psychosis and to encourage further research in this direction.

A nasal chemosensation-dependent critical window for somatosensory development

Nasal chemosensation is considered the evolutionarily oldest mammalian sense and, together with somatosensation, is crucial for neonatal well-being before auditory and visual pathways start engaging the brain. Using anatomical and functional approaches in mice, we reveal that odor-driven activity propagates to a large part of the cortex during the first postnatal week and enhances whisker-evoked activation of primary whisker somatosensory cortex (wS1). This effect disappears in adult animals, in line with the loss of excitatory connectivity from olfactory cortex to wS1. By performing neonatal odor deprivation, followed by electrophysiological and behavioral work in adult animals, we identify a key transient regulation of nasal chemosensory information necessary for the development of wS1 sensory-driven dynamics and somatosensation. Our work uncovers a cross-modal critical window for nasal chemosensation-dependent somatosensory functional maturation.

Dynamic modulation of mouse thalamocortical visual activity by salient sounds

Visual responses of the primary visual cortex (V1) are altered by sound. Sound-driven behavioral arousal suggests that, in addition to direct inputs from the primary auditory cortex (A1), multiple other sources may shape V1 responses to sound. Here, we show in anesthetized mice that sound (white noise, ≥70dB) drives a biphasic modulation of V1 visually driven gamma-band activity, comprising fast-transient inhibitory and slow, prolonged excitatory (A1-independent) arousal-driven components. An analogous yet quicker modulation of the visual response also occurred earlier in the visual pathway, at the level of the dorsolateral geniculate nucleus (dLGN), where sound transiently inhibited the early phasic visual response and subsequently induced a prolonged increase in tonic spiking activity and gamma rhythmicity. Our results demonstrate that sound-driven modulations of visual activity are not exclusive to V1 and suggest that thalamocortical inputs from the dLGN to V1 contribute to shaping V1 visual response to sound.

Auditory cortex conveys non-topographic sound localization signals to visual cortex

Spatiotemporally congruent sensory stimuli are fused into a unified percept. The auditory cortex (AC) sends projections to the primary visual cortex (V1), which could provide signals for binding spatially corresponding audio-visual stimuli. However, whether AC inputs in V1 encode sound location remains unknown. Using two-photon axonal calcium imaging and a speaker array, we measured the auditory spatial information transmitted from AC to layer 1 of V1. AC conveys information about the location of ipsilateral and contralateral sound sources to V1. Sound location could be accurately decoded by sampling AC axons in V1, providing a substrate for making location-specific audiovisual associations. However, AC inputs were not retinotopically arranged in V1, and audio-visual modulations of V1 neurons did not depend on the spatial congruency of the sound and light stimuli. The non-topographic sound localization signals provided by AC might allow the association of specific audiovisual spatial patterns in V1 neurons.

Antipsychotic drugs selectively decorrelate long-range interactions in deep cortical layers

Psychosis is characterized by a diminished ability of the brain to distinguish externally driven activity patterns from self-generated activity patterns. Antipsychotic drugs are a class of small molecules with relatively broad binding affinity for a variety of neuromodulator receptors that, in humans, can prevent or ameliorate psychosis. How these drugs influence the function of cortical circuits, and in particular their ability to distinguish between externally and self-generated activity patterns, is still largely unclear. To have experimental control over self-generated sensory feedback, we used a virtual reality environment in which the coupling between movement and visual feedback can be altered. We then used widefield calcium imaging to determine the cell type-specific functional effects of antipsychotic drugs in mouse dorsal cortex under different conditions of visuomotor coupling. By comparing cell type-specific activation patterns between locomotion onsets that were experimentally coupled to self-generated visual feedback and locomotion onsets that were not coupled, we show that deep cortical layers were differentially activated in these two conditions. We then show that the antipsychotic drug clozapine disrupted visuomotor integration at locomotion onsets also primarily in deep cortical layers. Given that one of the key components of visuomotor integration in cortex is long-range cortico-cortical connections, we tested whether the effect of clozapine was detectable in the correlation structure of activity patterns across dorsal cortex. We found that clozapine as well as two other antipsychotic drugs, aripiprazole and haloperidol, resulted in a strong reduction in correlations of layer 5 activity between cortical areas and impaired the spread of visuomotor prediction errors generated in visual cortex. Our results are consistent with the interpretation that a major functional effect of antipsychotic drugs is a selective alteration of long-range layer 5-mediated communication.

Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex

Primary sensory cortices respond to crossmodal stimuli-for example, auditory responses are found in primary visual cortex (V1). However, it remains unclear whether these responses reflect sensory inputs or behavioral modulation through sound-evoked body movement. We address this controversy by showing that sound-evoked activity in V1 of awake mice can be dissociated into auditory and behavioral components with distinct spatiotemporal profiles. The auditory component began at approximately 27 ms, was found in superficial and deep layers and originated from auditory cortex. Sound-evoked orofacial movements correlated with V1 neural activity starting at approximately 80-100 ms and explained auditory frequency tuning. Visual, auditory and motor activity were expressed by different laminar profiles and largely segregated subsets of neuronal populations. During simultaneous audiovisual stimulation, visual representations remained dissociable from auditory-related and motor-related activity. This three-fold dissociability of auditory, motor and visual processing is central to understanding how distinct inputs to visual cortex interact to support vision.

Rodents' visual gamma as a biomarker of pathological neural conditions

Neural gamma oscillations (indicatively 30-100 Hz) are ubiquitous: they are associated with a broad range of functions in multiple cortical areas and across many animal species. Experimental and computational works established gamma rhythms as a global emergent property of neuronal networks generated by the balanced and coordinated interaction of excitation and inhibition. Coherently, gamma activity is strongly influenced by the alterations of synaptic dynamics which are often associated with pathological neural dysfunctions. We argue therefore that these oscillations are an optimal biomarker for probing the mechanism of cortical dysfunctions. Gamma oscillations are also highly sensitive to external stimuli in sensory cortices, especially the primary visual cortex (V1), where the stimulus dependence of gamma oscillations has been thoroughly investigated. Gamma manipulation by visual stimuli tuning is particularly easy in rodents, which have become a standard animal model for investigating the effects of network alterations on gamma oscillations. Overall, gamma in the rodents' visual cortex offers an accessible probe on dysfunctional information processing in pathological conditions. Beyond vision-related dysfunctions, alterations of gamma oscillations in rodents were indeed also reported in neural deficits such as migraine, epilepsy and neurodegenerative or neuropsychiatric conditions such as Alzheimer's, schizophrenia and autism spectrum disorders. Altogether, the connections between visual cortical gamma activity and physio-pathological conditions in rodent models underscore the potential of gamma oscillations as markers of neuronal (dys)functioning.

Behavior-relevant top-down cross-modal predictions in mouse neocortex

Animals adapt to a constantly changing world by predicting their environment and the consequences of their actions. The predictive coding hypothesis proposes that the brain generates predictions and continuously compares them with sensory inputs to guide behavior. However, how the brain reconciles conflicting top-down predictions and bottom-up sensory information remains unclear. To address this question, we simultaneously imaged neuronal populations in the mouse somatosensory barrel cortex and posterior parietal cortex during an auditory-cued texture discrimination task. In mice that had learned the task with fixed tone-texture matching, the presentation of mismatched pairing induced conflicts between tone-based texture predictions and actual texture inputs. When decisions were based on the predicted rather than the actual texture, top-down information flow was dominant and texture representations in both areas were modified, whereas dominant bottom-up information flow led to correct representations and behavioral choice. Our findings provide evidence for hierarchical predictive coding in the mouse neocortex.

The locus coeruleus as a global model failure system

Predictive processing models posit that brains constantly attempt to predict their sensory inputs. Prediction errors signal when these predictions are incorrect and are thought to be instructive signals that drive corrective plasticity. Recent findings support the idea that the locus coeruleus (LC) - a brain-wide neuromodulatory system - signals several types of prediction error. I discuss how these findings support models proposing that the LC signals global model failures: instances where predictions about the world are strongly violated. Focusing on the cortex, I explore the utility of this signal in learning rate control, how the LC circuit may compute the signal, and how this view may aid our understanding of neurodivergence.

Functional neuronal circuits emerge in the absence of developmental activity

The complex neuronal circuitry of the brain develops from limited information contained in the genome. After the genetic code instructs the birth of neurons, the emergence of brain regions, and the formation of axon tracts, it is believed that temporally structured spiking activity shapes circuits for behavior. Here, we challenge the learning-dominated assumption that spiking activity is required for circuit formation by quantifying its contribution to the development of visually-guided swimming in the larval zebrafish. We found that visual experience had no effect on the emergence of the optomotor response (OMR) in dark-reared zebrafish. We then raised animals while pharmacologically silencing action potentials with the sodium channel blocker tricaine. After washout of the anesthetic, fish could swim and performed with 75-90% accuracy in the OMR paradigm. Brain-wide imaging confirmed that neuronal circuits came 'online' fully tuned, without requiring activity-dependent plasticity. Thus, complex sensory-guided behaviors can emerge through activity-independent developmental mechanisms.

Predictive neural computations in the cerebellum contribute to motor planning and faster behavioral responses in larval zebrafish

The ability to predict the future based on past experience lies at the core of the brain's ability to adapt behavior. However, the neural mechanisms that participate in generating and updating predictions are not clearly understood. Further, the evolutionary antecedents and the prevalence of predictive processing among vertebrates are even less explored. Here, we show evidence of predictive processing via the involvement of cerebellar circuits in larval zebrafish. We presented stereotyped optic flow stimuli to larval zebrafish to evoke swims and discovered that lesioning the cerebellum abolished prediction-dependent modulation of swim latency. When expectations of optic flow direction did not match with reality, error signals arrive at Purkinje cells via the olivary climbing fibers, whereas granule cells and Purkinje cells encode signals of expectation. Strong neural representations of expectation correlate with faster swim responses and vice versa. In sum, our results show evidence for predictive processing in nonmammalian vertebrates with the involvement of cerebellum, an evolutionarily conserved brain structure.

How deep is the brain? The shallow brain hypothesis

Deep learning and predictive coding architectures commonly assume that inference in neural networks is hierarchical. However, largely neglected in deep learning and predictive coding architectures is the neurobiological evidence that all hierarchical cortical areas, higher or lower, project to and receive signals directly from subcortical areas. Given these neuroanatomical facts, today's dominance of cortico-centric, hierarchical architectures in deep learning and predictive coding networks is highly questionable; such architectures are likely to be missing essential computational principles the brain uses. In this Perspective, we present the shallow brain hypothesis: hierarchical cortical processing is integrated with a massively parallel process to which subcortical areas substantially contribute. This shallow architecture exploits the computational capacity of cortical microcircuits and thalamo-cortical loops that are not included in typical hierarchical deep learning and predictive coding networks. We argue that the shallow brain architecture provides several critical benefits over deep hierarchical structures and a more complete depiction of how mammalian brains achieve fast and flexible computational capabilities.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

How 'visual' is the visual cortex? The interactions between the visual cortex and other sensory, motivational and motor systems as enabling factors for visual perception

The definition of the visual cortex is primarily based on the evidence that lesions of this area impair visual perception. However, this does not exclude that the visual cortex may process more information than of retinal origin alone, or that other brain structures contribute to vision. Indeed, research across the past decades has shown that non-visual information, such as neural activity related to reward expectation and value, locomotion, working memory and other sensory modalities, can modulate primary visual cortical responses to retinal inputs. Nevertheless, the function of this non-visual information is poorly understood. Here we review recent evidence, coming primarily from studies in rodents, arguing that non-visual and motor effects in visual cortex play a role in visual processing itself, for instance disentangling direct auditory effects on visual cortex from effects of sound-evoked orofacial movement. These findings are placed in a broader framework casting vision in terms of predictive processing under control of frontal, reward- and motor-related systems. In contrast to the prevalent notion that vision is exclusively constructed by the visual cortical system, we propose that visual percepts are generated by a larger network-the extended visual system-spanning other sensory cortices, supramodal areas and frontal systems. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Robust multisensory deviance detection in the mouse parietal associative area

Context modulates how information is processed in the mammalian brain. For example, brain responses are amplified to contextually unusual stimuli. This phenomenon, known as "deviance detection,"1,2 is well documented in early, primary sensory cortex, where large responses are generated to simple stimuli that deviate from their context in low-order properties, such as line orientation, size, or pitch.2,3,4,5 However, the extent to which neural deviance detection manifests (1) in broader cortical networks and (2) to simple versus complex stimuli, which deviate only in their higher-order, multisensory properties, is not known. Consistent with a predictive processing framework,6,7 we hypothesized that deviance detection manifests in a hierarchical manner across cortical networks,8,9 emerging later and further downstream when stimulus deviance is complex. To test this, we examined brain responses of awake mice to simple unisensory deviants (e.g., visual line gratings, deviating from context in their orientation alone) versus complex multisensory deviants (i.e., audiovisual pairs, deviating from context only in their audiovisual pairing but not visual or auditory content alone). We find that mouse parietal associative area-a higher cortical region-displays robust multisensory deviance detection. In contrast, primary visual cortex exhibits strong unisensory visual deviance detection but weaker multisensory deviance detection. These results suggest that deviance detection signals in the cortex may be conceptualized as "prediction errors," which are primarily fed forward-or downstream-in cortical networks.6,7.

Molecularly targetable cell types in mouse visual cortex have distinguishable prediction error responses

Predictive processing postulates the existence of prediction error neurons in cortex. Neurons with both negative and positive prediction error response properties have been identified in layer 2/3 of visual cortex, but whether they correspond to transcriptionally defined subpopulations is unclear. Here we used the activity-dependent, photoconvertible marker CaMPARI2 to tag neurons in layer 2/3 of mouse visual cortex during stimuli and behaviors designed to evoke prediction errors. We performed single-cell RNA-sequencing on these populations and found that previously annotated Adamts2 and Rrad layer 2/3 transcriptional cell types were enriched when photolabeling during stimuli that drive negative or positive prediction error responses, respectively. Finally, we validated these results functionally by designing artificial promoters for use in AAV vectors to express genetically encoded calcium indicators. Thus, transcriptionally distinct cell types in layer 2/3 that can be targeted using AAV vectors exhibit distinguishable negative and positive prediction error responses.

Fast updating feedback from piriform cortex to the olfactory bulb relays multimodal reward contingency signals during rule-reversal

While animals readily adjust their behavior to adapt to relevant changes in the environment, the neural pathways enabling these changes remain largely unknown. Here, using multiphoton imaging, we investigated whether feedback from the piriform cortex to the olfactory bulb supports such behavioral flexibility. To this end, we engaged head-fixed mice in a multimodal rule-reversal task guided by olfactory and auditory cues. Both odor and, surprisingly, the sound cues triggered cortical bulbar feedback responses which preceded the behavioral report. Responses to the same sensory cue were strongly modulated upon changes in stimulus-reward contingency (rule reversals). The re-shaping of individual bouton responses occurred within seconds of the rule-reversal events and was correlated with changes in the behavior. Optogenetic perturbation of cortical feedback within the bulb disrupted the behavioral performance. Our results indicate that the piriform-to-olfactory bulb feedback carries reward contingency signals and is rapidly re-formatted according to changes in the behavioral context.

Multimodal Temporal Pattern Discrimination Is Encoded in Visual Cortical Dynamics

Discriminating between temporal features in sensory stimuli is critical to complex behavior and decision-making. However, how sensory cortical circuit mechanisms contribute to discrimination between subsecond temporal components in sensory events is unclear. To elucidate the mechanistic underpinnings of timing in primary visual cortex (V1), we recorded from V1 using two-photon calcium imaging in awake-behaving mice performing a go/no-go discrimination timing task, which was composed of patterns of subsecond audiovisual stimuli. In both conditions, activity during the early stimulus period was temporally coordinated with the preferred stimulus. However, while network activity increased in the preferred condition, network activity was increasingly suppressed in the nonpreferred condition over the stimulus period. Multiple levels of analyses suggest that discrimination between subsecond intervals that are contained in rhythmic patterns can be accomplished by local neural dynamics in V1.

Spike-based coupling between single neurons and populations across rat sensory cortices, perirhinal cortex, and hippocampus

Cortical computations require coordination of neuronal activity within and across multiple areas. We characterized spiking relationships within and between areas by quantifying coupling of single neurons to population firing patterns. Single-neuron population coupling (SNPC) was investigated using ensemble recordings from hippocampal CA1 region and somatosensory, visual, and perirhinal cortices. Within-area coupling was heterogeneous across structures, with area CA1 showing higher levels than neocortical regions. In contrast to known anatomical connectivity, between-area coupling showed strong firing coherence of sensory neocortices with CA1, but less with perirhinal cortex. Cells in sensory neocortices and CA1 showed positive correlations between within- and between-area coupling; these were weaker for perirhinal cortex. All four areas harbored broadcasting cells, connecting to multiple external areas, which was uncorrelated to within-area coupling strength. When examining correlations between SNPC and spatial coding, we found that, if such correlations were significant, they were negative. This result was consistent with an overall preservation of SNPC across different brain states, suggesting a strong dependence on intrinsic network connectivity. Overall, SNPC offers an important window on cell-to-population synchronization in multi-area networks. Instead of pointing to specific information-coding functions, our results indicate a primary function of SNPC in dynamically organizing communication in systems composed of multiple, interconnected areas.

Dual-stream cortical pathways mediate sensory prediction

Predictions are constantly generated from diverse sources to optimize cognitive functions in the ever-changing environment. However, the neural origin and generation process of top-down induced prediction remain elusive. We hypothesized that motor-based and memory-based predictions are mediated by distinct descending networks from motor and memory systems to the sensory cortices. Using functional magnetic resonance imaging (fMRI) and a dual imagery paradigm, we found that motor and memory upstream systems activated the auditory cortex in a content-specific manner. Moreover, the inferior and posterior parts of the parietal lobe differentially relayed predictive signals in motor-to-sensory and memory-to-sensory networks. Dynamic causal modeling of directed connectivity revealed selective enabling and modulation of connections that mediate top-down sensory prediction and ground the distinctive neurocognitive basis of predictive processing.

Impulse perturbation reveals cross-modal access to sensory working memory through learned associations

We investigated if learned associations between visual and auditory stimuli can afford full cross-modal access to working memory. Previous research using the impulse perturbation technique has shown that cross-modal access to working memory is one-sided; visual impulses reveal both auditory and visual memoranda, but auditory impulses do not seem to reveal visual memoranda (Wolff et al., 2020b). Our participants first learned to associate six auditory pure tones with six visual orientation gratings. Next, a delayed match-to-sample task for the orientations was completed, while EEG was recorded. Orientation memories were recalled either via their learned auditory counterpart, or were visually presented. We then decoded the orientation memories from the EEG responses to both auditory and visual impulses presented during the memory delay. Working memory content could always be decoded from visual impulses. Importantly, through recall of the learned associations, the auditory impulse also evoked a decodable response from the visual WM network, providing evidence for full cross-modal access. We also observed that after a brief initial dynamic period, the representational codes of the memory items generalized across time, as well as between perceptual maintenance and long-term recall conditions. Our results thus demonstrate that accessing learned associations in long-term memory provides a cross-modal pathway to working memory that seems to be based on a common coding scheme.

Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex

Crossmodal correspondences describe our tendency to associate sensory features from different modalities with each other, such as the pitch of a sound with the size of a visual object. While such crossmodal correspondences (or associations) are described in many behavioural studies their neurophysiological correlates remain unclear. Under the current working model of multisensory perception both a low- and a high-level account seem plausible. That is, the neurophysiological processes shaping these associations could commence in low-level sensory regions, or may predominantly emerge in high-level association regions of semantic and object identification networks. We exploited steady-state visual evoked potentials (SSVEP) to directly probe this question, focusing on the associations between pitch and the visual features of size, hue or chromatic saturation. We found that SSVEPs over occipital regions are sensitive to the congruency between pitch and size, and a source analysis pointed to an origin around primary visual cortices. We speculate that this signature of the pitch-size association in low-level visual cortices reflects the successful pairing of congruent visual and acoustic object properties and may contribute to establishing causal relations between multisensory objects. Besides this, our study also provides a paradigm can be exploited to study other crossmodal associations involving visual stimuli in the future.

Locomotion-induced gain of visual responses cannot explain visuomotor mismatch responses in layer 2/3 of primary visual cortex

The aim of this work is to provide a comment on a recent paper by Muzzu and Saleem (2021), which claims that visuomotor mismatch responses in mouse visual cortex can be explained by a locomotion-induced gain of visual halt responses. Our primary concern is that without directly comparing these responses with mismatch responses, the claim that one response can explain the other appears difficult to uphold, more so because previous work finds that a uniform locomotion-induced gain cannot explain mismatch responses. To support these arguments, we analyze layer 2/3 calcium imaging datasets and show that coupling between visual flow and locomotion greatly enhances mismatch responses in an experience-dependent manner compared with halts in non-coupled visual flow. This is consistent with mismatch responses representing visuomotor prediction errors. Thus, we conclude that while feature selectivity might contribute to mismatch responses in mouse visual cortex, it cannot explain these responses.

Behavioral origin of sound-evoked activity in mouse visual cortex

Sensory cortices can be affected by stimuli of multiple modalities and are thus increasingly thought to be multisensory. For instance, primary visual cortex (V1) is influenced not only by images but also by sounds. Here we show that the activity evoked by sounds in V1, measured with Neuropixels probes, is stereotyped across neurons and even across mice. It is independent of projections from auditory cortex and resembles activity evoked in the hippocampal formation, which receives little direct auditory input. Its low-dimensional nature starkly contrasts the high-dimensional code that V1 uses to represent images. Furthermore, this sound-evoked activity can be precisely predicted by small body movements that are elicited by each sound and are stereotyped across trials and mice. Thus, neural activity that is apparently multisensory may simply arise from low-dimensional signals associated with internal state and behavior.

Magnetoencephalography recordings reveal the neural mechanisms of auditory contributions to improved visual detection

Sounds enhance the detection of visual stimuli while concurrently biasing an observer's decisions. To investigate the neural mechanisms that underlie such multisensory interactions, we decoded time-resolved Signal Detection Theory sensitivity and criterion parameters from magneto-encephalographic recordings of participants that performed a visual detection task. We found that sounds improved visual detection sensitivity by enhancing the accumulation and maintenance of perceptual evidence over time. Meanwhile, criterion decoding analyses revealed that sounds induced brain activity patterns that resembled the patterns evoked by an actual visual stimulus. These two complementary mechanisms of audiovisual interplay differed in terms of their automaticity: Whereas the sound-induced enhancement in visual sensitivity depended on participants being actively engaged in a detection task, we found that sounds activated the visual cortex irrespective of task demands, potentially inducing visual illusory percepts. These results challenge the classical assumption that sound-induced increases in false alarms exclusively correspond to decision-level biases.

Where is the error? Hierarchical predictive coding through dendritic error computation

Top-down feedback in cortex is critical for guiding sensory processing, which has prominently been formalized in the theory of hierarchical predictive coding (hPC). However, experimental evidence for error units, which are central to the theory, is inconclusive and it remains unclear how hPC can be implemented with spiking neurons. To address this, we connect hPC to existing work on efficient coding in balanced networks with lateral inhibition and predictive computation at apical dendrites. Together, this work points to an efficient implementation of hPC with spiking neurons, where prediction errors are computed not in separate units, but locally in dendritic compartments. We then discuss the correspondence of this model to experimentally observed connectivity patterns, plasticity, and dynamics in cortex.

The role of the salience network in cognitive and affective deficits

Analysis and interpretation of studies on cognitive and affective dysregulation often draw upon the network paradigm, especially the Triple Network Model, which consists of the default mode network (DMN), the frontoparietal network (FPN), and the salience network (SN). DMN activity is primarily dominant during cognitive leisure and self-monitoring processes. The FPN peaks during task involvement and cognitive exertion. Meanwhile, the SN serves as a dynamic "switch" between the DMN and FPN, in line with salience and cognitive demand. In the cognitive and affective domains, dysfunctions involving SN activity are connected to a broad spectrum of deficits and maladaptive behavioral patterns in a variety of clinical disorders, such as depression, insomnia, narcissism, PTSD (in the case of SN hyperactivity), chronic pain, and anxiety, high degrees of neuroticism, schizophrenia, epilepsy, autism, and neurodegenerative illnesses, bipolar disorder (in the case of SN hypoactivity). We discuss behavioral and neurological data from various research domains and present an integrated perspective indicating that these conditions can be associated with a widespread disruption in predictive coding at multiple hierarchical levels. We delineate the fundamental ideas of the brain network paradigm and contrast them with the conventional modular method in the first section of this article. Following this, we outline the interaction model of the key functional brain networks and highlight recent studies coupling SN-related dysfunctions with cognitive and affective impairments.

Probing top-down information in neocortical layer 1

Accurate perception of the environment is a constructive process that requires integration of external bottom-up sensory signals with internally generated top-down information. Decades of work have elucidated how sensory neocortex processes physical stimulus features. By contrast, examining how top-down information is encoded and integrated with bottom-up signals has been challenging using traditional neuroscience methods. Recent technological advances in functional imaging of brain-wide afferents in behaving mice have enabled the direct measurement of top-down information. Here, we review the emerging literature on encoding of these internally generated signals by different projection systems enriched in neocortical layer 1 during defined brain functions, including memory, attention, and predictive coding. Moreover, we identify gaps in current knowledge and highlight future directions for this rapidly advancing field.

Precise movement-based predictions in the mouse auditory cortex

Many of the sensations experienced by an organism are caused by their own actions, and accurately anticipating both the sensory features and timing of self-generated stimuli is crucial to a variety of behaviors. In the auditory cortex, neural responses to self-generated sounds exhibit frequency-specific suppression, suggesting that movement-based predictions may be implemented early in sensory processing. However, it remains unknown whether this modulation results from a behaviorally specific and temporally precise prediction, nor is it known whether corresponding expectation signals are present locally in the auditory cortex. To address these questions, we trained mice to expect the precise acoustic outcome of a forelimb movement using a closed-loop sound-generating lever. Dense neuronal recordings in the auditory cortex revealed suppression of responses to self-generated sounds that was specific to the expected acoustic features, to a precise position within the movement, and to the movement that was coupled to sound during training. Prediction-based suppression was concentrated in L2/3 and L5, where deviations from expectation also recruited a population of prediction-error neurons that was otherwise unresponsive. Recording in the absence of sound revealed abundant movement signals in deep layers that were biased toward neurons tuned to the expected sound, as well as expectation signals that were present throughout the cortex and peaked at the time of expected auditory feedback. Together, these findings identify distinct populations of auditory cortical neurons with movement, expectation, and error signals consistent with a learned internal model linking an action to its specific acoustic outcome.

Changing the Tendency to Integrate the Senses

Integration of sensory signals that emanate from the same source, such as the visual of lip articulations and the sound of the voice of a speaking individual, can improve perception of the source signal (e.g., speech). Because momentary sensory inputs are typically corrupted with internal and external noise, there is almost always a discrepancy between the inputs, facing the perceptual system with the problem of determining whether the two signals were caused by the same source or different sources. Thus, whether or not multisensory stimuli are integrated and the degree to which they are bound is influenced by factors such as the prior expectation of a common source. We refer to this factor as the tendency to bind stimuli, or for short, binding tendency. In theory, the tendency to bind sensory stimuli can be learned by experience through the acquisition of the probabilities of the co-occurrence of the stimuli. It can also be influenced by cognitive knowledge of the environment. The binding tendency varies across individuals and can also vary within an individual over time. Here, we review the studies that have investigated the plasticity of binding tendency. We discuss the protocols that have been reported to produce changes in binding tendency, the candidate learning mechanisms involved in this process, the possible neural correlates of binding tendency, and outstanding questions pertaining to binding tendency and its plasticity. We conclude by proposing directions for future research and argue that understanding mechanisms and recipes for increasing binding tendency can have important clinical and translational applications for populations or individuals with a deficiency in multisensory integration.

Mystical Experience

This paper proposes to study mystical experience by contrasting it with “ordinary” experience, i.e., with standard consciousness. It emphasises the construed nature of standard consciousness and the role that the mutual connectedness of mental contents plays in its construction. It then shows that removal of the factors that are responsible for the “making” of standard consciousness accounts for the principal features of mystical experience; these features are, therefore, mainly negative. Understanding mystical experience as the suppression of factors that contribute to the construction of standard consciousness, along with a discussion of the mechanism that makes this possible, permits answers to some frequently asked questions, such as: Why is mystical experience ineffable? What is its epistemic status? Does it have implications for our understanding of mind, consciousness, and self?

Estimating How Sounds Modulate Orientation Representation in the Primary Visual Cortex Using Shallow Neural Networks

Audiovisual perception results from the interaction between visual and auditory processing. Hence, presenting auditory and visual inputs simultaneously usually improves the accuracy of the unimodal percepts, but can also lead to audiovisual illusions. Cross-talks between visual and auditory inputs during sensory processing were recently shown to occur as early as in the primary visual cortex (V1). In a previous study, we demonstrated that sounds improve the representation of the orientation of visual stimuli in the naïve mouse V1 by promoting the recruitment of neurons better tuned to the orientation and direction of the visual stimulus. However, we did not test if this type of modulation was still present when the auditory and visual stimuli were both behaviorally relevant. To determine the effect of sounds on active visual processing, we performed calcium imaging in V1 while mice were performing an audiovisual task. We then compared the representations of the task stimuli orientations in the unimodal visual and audiovisual context using shallow neural networks (SNNs). SNNs were chosen because of the biological plausibility of their computational structure and the possibility of identifying post hoc the biological neurons having the strongest influence on the classification decision. We first showed that SNNs can categorize the activity of V1 neurons evoked by drifting gratings of 12 different orientations. Then, we demonstrated using the connection weight approach that SNN training assigns the largest computational weight to the V1 neurons having the best orientation and direction selectivity. Finally, we showed that it is possible to use SNNs to determine how V1 neurons represent the orientations of stimuli that do not belong to the set of orientations used for SNN training. Once the SNN approach was established, we replicated the previous finding that sounds improve orientation representation in the V1 of naïve mice. Then, we showed that, in mice performing an audiovisual detection task, task tones improve the representation of the visual cues associated with the reward while deteriorating the representation of non-rewarded cues. Altogether, our results suggest that the direction of sound modulation in V1 depends on the behavioral relevance of the visual cue.

Multisensory Integration: Is Medial Prefrontal Cortex Signaling Relevant for the Treatment of Higher-Order Visual Dysfunctions?

In the mammalian brain, information processing in sensory modalities and global mechanisms of multisensory integration facilitate perception. Emerging experimental evidence suggests that the contribution of multisensory integration to sensory perception is far more complex than previously expected. Here we revise how associative areas such as the prefrontal cortex, which receive and integrate inputs from diverse sensory modalities, can affect information processing in unisensory systems via processes of down-stream signaling. We focus our attention on the influence of the medial prefrontal cortex on the processing of information in the visual system and whether this phenomenon can be clinically used to treat higher-order visual dysfunctions. We propose that non-invasive and multisensory stimulation strategies such as environmental enrichment and/or attention-related tasks could be of clinical relevance to fight cerebral visual impairment.