Efficient and adaptive sensory codes

Memristor-based adaptive neuromorphic perception in unstructured environments

Efficient operation of control systems in robotics or autonomous driving targeting real-world navigation scenarios requires perception methods that allow them to understand and adapt to unstructured environments with good accuracy, adaptation, and generality, similar to humans. To address this need, we present a memristor-based differential neuromorphic computing, perceptual signal processing, and online adaptation method providing neuromorphic style adaptation to external sensory stimuli. The adaptation ability and generality of this method are confirmed in two application scenarios: object grasping and autonomous driving. In the former, a robot hand realizes safe and stable grasping through fast ( ~ 1 ms) adaptation based on the tactile object features with a single memristor. In the latter, decision-making information of 10 unstructured environments in autonomous driving is extracted with an accuracy of 94% with a 40×25 memristor array. By mimicking human low-level perception mechanisms, the electronic neuromorphic circuit-based method achieves real-time adaptation and high-level reactions to unstructured environments.

Structure, dynamics, coding and optimal biophysical parameters of efficient excitatory-inhibitory spiking networks

The principle of efficient coding posits that sensory cortical networks are designed to encode maximal sensory information with minimal metabolic cost. Despite the major influence of efficient coding in neuro-science, it has remained unclear whether fundamental empirical properties of neural network activity can be explained solely based on this normative principle. Here, we rigorously derive the structural, coding, biophysical and dynamical properties of excitatory-inhibitory recurrent networks of spiking neurons that emerge directly from imposing that the network minimizes an instantaneous loss function and a time-averaged performance measure enacting efficient coding. The optimal network has biologically-plausible biophysical features, including realistic integrate-and-fire spiking dynamics, spike-triggered adaptation, and a non-stimulus-specific excitatory external input regulating metabolic cost. The efficient network has excitatory-inhibitory recurrent connectivity between neurons with similar stimulus tuning implementing feature-specific competition, similar to that recently found in visual cortex. Networks with unstructured connectivity cannot reach comparable levels of coding efficiency. The optimal biophysical parameters include 4 to 1 ratio of excitatory vs inhibitory neurons and 3 to 1 ratio of mean inhibitory-to-inhibitory vs. excitatory-to-inhibitory connectivity that closely match those of cortical sensory networks. The efficient network has biologically-plausible spiking dynamics, with a tight instantaneous E-I balance that makes them capable to achieve efficient coding of external stimuli varying over multiple time scales. Together, these results explain how efficient coding may be implemented in cortical networks and suggests that key properties of biological neural networks may be accounted for by efficient coding.

Encoding surprise by retinal ganglion cells

The efficient coding hypothesis posits that early sensory neurons transmit maximal information about sensory stimuli, given internal constraints. A central prediction of this theory is that neurons should preferentially encode stimuli that are most surprising. Previous studies suggest this may be the case in early visual areas, where many neurons respond strongly to rare or surprising stimuli. For example, previous research showed that when presented with a rhythmic sequence of full-field flashes, many retinal ganglion cells (RGCs) respond strongly at the instance the flash sequence stops, and when another flash would be expected. This phenomenon is called the 'omitted stimulus response'. However, it is not known whether the responses of these cells varies in a graded way depending on the level of stimulus surprise. To investigate this, we presented retinal neurons with extended sequences of stochastic flashes. With this stimulus, the surprise associated with a particular flash/silence, could be quantified analytically, and varied in a graded manner depending on the previous sequences of flashes and silences. Interestingly, we found that RGC responses could be well explained by a simple normative model, which described how they optimally combined their prior expectations and recent stimulus history, so as to encode surprise. Further, much of the diversity in RGC responses could be explained by the model, due to the different prior expectations that different neurons had about the stimulus statistics. These results suggest that even as early as the retina many cells encode surprise, relative to their own, internally generated expectations.

A unifying theory explains seemingly contradictory biases in perceptual estimation

Perceptual biases are widely regarded as offering a window into the neural computations underlying perception. To understand these biases, previous work has proposed a number of conceptually different, and even seemingly contradictory, explanations, including attraction to a Bayesian prior, repulsion from the prior due to efficient coding and central tendency effects on a bounded range. We present a unifying Bayesian theory of biases in perceptual estimation derived from first principles. We demonstrate theoretically an additive decomposition of perceptual biases into attraction to a prior, repulsion away from regions with high encoding precision and regression away from the boundary. The results reveal a simple and universal rule for predicting the direction of perceptual biases. Our theory accounts for, and yields, new insights regarding biases in the perception of a variety of stimulus attributes, including orientation, color and magnitude. These results provide important constraints on the neural implementations of Bayesian computations.

State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways

The nervous system uses fast- and slow-adapting sensory detectors in parallel to enable neuronal representations of external states and their temporal dynamics. It is unknown whether this dichotomy also applies to internal representations that have no direct correlation in the physical world. Here we find that two distinct dopamine (DA) neuron subtypes encode either a state or its rate-of-change. In mice performing a reward-seeking task, we found that the animal's behavioral state and rate-of-change were encoded by the sustained activity of DA neurons in medial ventral tegmental area (VTA) DA neurons and transient activity in lateral VTA DA neurons, respectively. The neural activity patterns of VTA DA cell bodies matched DA release patterns within anatomically defined mesoaccumbal pathways. Based on these results, we propose a model in which the DA system uses two parallel lines for proportional-differential encoding of a state variable and its temporal dynamics.

Two Sides of the Same Coin: Efficient and Predictive Neural Coding

Some visual properties are consistent across a wide range of environments, while other properties are more labile. The efficient coding hypothesis states that many of these regularities in the environment can be discarded from neural representations, thus allocating more of the brain's dynamic range to properties that are likely to vary. This paradigm is less clear about how the visual system prioritizes different pieces of information that vary across visual environments. One solution is to prioritize information that can be used to predict future events, particularly those that guide behavior. The relationship between the efficient coding and future prediction paradigms is an area of active investigation. In this review, we argue that these paradigms are complementary and often act on distinct components of the visual input. We also discuss how normative approaches to efficient coding and future prediction can be integrated.

Dynamics of cortical contrast adaptation predict perception of signals in noise

Neurons throughout the sensory pathway adapt their responses depending on the statistical structure of the sensory environment. Contrast gain control is a form of adaptation in the auditory cortex, but it is unclear whether the dynamics of gain control reflect efficient adaptation, and whether they shape behavioral perception. Here, we trained mice to detect a target presented in background noise shortly after a change in the contrast of the background. The observed changes in cortical gain and behavioral detection followed the dynamics of a normative model of efficient contrast gain control; specifically, target detection and sensitivity improved slowly in low contrast, but degraded rapidly in high contrast. Auditory cortex was required for this task, and cortical responses were not only similarly affected by contrast but predicted variability in behavioral performance. Combined, our results demonstrate that dynamic gain adaptation supports efficient coding in auditory cortex and predicts the perception of sounds in noise.

Panoramic visual statistics shape retina-wide organization of receptive fields

Statistics of natural scenes are not uniform-their structure varies dramatically from ground to sky. It remains unknown whether these nonuniformities are reflected in the large-scale organization of the early visual system and what benefits such adaptations would confer. Here, by relying on the efficient coding hypothesis, we predict that changes in the structure of receptive fields across visual space increase the efficiency of sensory coding. Using the mouse (Mus musculus) as a model species, we show that receptive fields of retinal ganglion cells change their shape along the dorsoventral retinal axis, with a marked surround asymmetry at the visual horizon, in agreement with our predictions. Our work demonstrates that, according to principles of efficient coding, the panoramic structure of natural scenes is exploited by the retina across space and cell types.

Computational methods to study information processing in neural circuits

The brain is an information processing machine and thus naturally lends itself to be studied using computational tools based on the principles of information theory. For this reason, computational methods based on or inspired by information theory have been a cornerstone of practical and conceptual progress in neuroscience. In this Review, we address how concepts and computational tools related to information theory are spurring the development of principled theories of information processing in neural circuits and the development of influential mathematical methods for the analyses of neural population recordings. We review how these computational approaches reveal mechanisms of essential functions performed by neural circuits. These functions include efficiently encoding sensory information and facilitating the transmission of information to downstream brain areas to inform and guide behavior. Finally, we discuss how further progress and insights can be achieved, in particular by studying how competing requirements of neural encoding and readout may be optimally traded off to optimize neural information processing.

Calibrating Vision: Concepts and Questions

The idea that visual coding and perception are shaped by experience and adjust to changes in the environment or the observer is universally recognized as a cornerstone of visual processing, yet the functions and processes mediating these calibrations remain in many ways poorly understood. In this article we review a number of facets and issues surrounding the general notion of calibration, with a focus on plasticity within the encoding and representational stages of visual processing. These include how many types of calibrations there are - and how we decide; how plasticity for encoding is intertwined with other principles of sensory coding; how it is instantiated at the level of the dynamic networks mediating vision; how it varies with development or between individuals; and the factors that may limit the form or degree of the adjustments. Our goal is to give a small glimpse of an enormous and fundamental dimension of vision, and to point to some of the unresolved questions in our understanding of how and why ongoing calibrations are a pervasive and essential element of vision.

Divisive normalization is an efficient code for multivariate Pareto-distributed environments

Divisive normalization is a canonical computation in the brain, observed across neural systems, that is often considered to be an implementation of the efficient coding principle. We provide a theoretical result that makes the conditions under which divisive normalization is an efficient code analytically precise: We show that, in a low-noise regime, encoding an n-dimensional stimulus via divisive normalization is efficient if and only if its prevalence in the environment is described by a multivariate Pareto distribution. We generalize this multivariate analog of histogram equalization to allow for arbitrary metabolic costs of the representation, and show how different assumptions on costs are associated with different shapes of the distributions that divisive normalization efficiently encodes. Our result suggests that divisive normalization may have evolved to efficiently represent stimuli with Pareto distributions. We demonstrate that this efficiently encoded distribution is consistent with stylized features of naturalistic stimulus distributions such as their characteristic conditional variance dependence, and we provide empirical evidence suggesting that it may capture the statistics of filter responses to naturalistic images. Our theoretical finding also yields empirically testable predictions across sensory domains on how the divisive normalization parameters should be tuned to features of the input distribution.

Efficient information coding and degeneracy in the nervous system

Efficient information coding (EIC) is a universal biological framework rooted in the fundamental principle that system responses should match their natural stimulus statistics for maximizing environmental information. Quantitatively assessed through information theory, such adaptation to the environment occurs at all biological levels and timescales. The context dependence of environmental stimuli and the need for stable adaptations make EIC a daunting task. We argue that biological complexity is the principal architect that subserves deft execution of stable EIC. Complexity in a system is characterized by several functionally segregated subsystems that show a high degree of functional integration when they interact with each other. Complex biological systems manifest heterogeneities and degeneracy, wherein structurally different subsystems could interact to yield the same functional outcome. We argue that complex systems offer several choices that effectively implement EIC and homeostasis for each of the different contexts encountered by the system.

The structures and functions of correlations in neural population codes

The collective activity of a population of neurons, beyond the properties of individual cells, is crucial for many brain functions. A fundamental question is how activity correlations between neurons affect how neural populations process information. Over the past 30 years, major progress has been made on how the levels and structures of correlations shape the encoding of information in population codes. Correlations influence population coding through the organization of pairwise-activity correlations with respect to the similarity of tuning of individual neurons, by their stimulus modulation and by the presence of higher-order correlations. Recent work has shown that correlations also profoundly shape other important functions performed by neural populations, including generating codes across multiple timescales and facilitating information transmission to, and readout by, downstream brain areas to guide behaviour. Here, we review this recent work and discuss how the structures of correlations can have opposite effects on the different functions of neural populations, thus creating trade-offs and constraints for the structure-function relationships of population codes. Further, we present ideas on how to combine large-scale simultaneous recordings of neural populations, computational models, analyses of behaviour, optogenetics and anatomy to unravel how the structures of correlations might be optimized to serve multiple functions.

Retinal Encoding of Natural Scenes

An ultimate goal in retina science is to understand how the neural circuit of the retina processes natural visual scenes. Yet most studies in laboratories have long been performed with simple, artificial visual stimuli such as full-field illumination, spots of light, or gratings. The underlying assumption is that the features of the retina thus identified carry over to the more complex scenario of natural scenes. As the application of corresponding natural settings is becoming more commonplace in experimental investigations, this assumption is being put to the test and opportunities arise to discover processing features that are triggered by specific aspects of natural scenes. Here, we review how natural stimuli have been used to probe, refine, and complement knowledge accumulated under simplified stimuli, and we discuss challenges and opportunities along the way toward a comprehensive understanding of the encoding of natural scenes. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Force Mapping Reveals the Spatial Distribution of Individual Proteins in a Neuron

Conventional methods for studying the spatial distribution and expression level of proteins within neurons have primarily relied on immunolabeling and/or signal amplification. Here, we present an atomic force microscopy (AFM)-based nanoscale force mapping method, where Anti-LIMK1-tethered AFM probes were used to visualize individual LIMK1 proteins in cultured neurons directly through force measurements. We observed that the number density of LIMK1 decreased in neuronal somas after the cells were depolarized. We also elucidated the spatial distribution of LIMK1 in single spine areas and found that the protein predominantly locates at heads of spines rather than dendritic shafts. The study demonstrates that our method enables unveiling of the abundance and spatial distribution of a protein of interest in neurons without signal amplification or labeling. We expected that this approach should facilitate the studies of protein expression phenomena in depth in a wide range of biological systems.

Optimal Population Coding for Dynamic Input by Nonequilibrium Networks

The efficient coding hypothesis states that neural response should maximize its information about the external input. Theoretical studies focus on optimal response in single neuron and population code in networks with weak pairwise interactions. However, more biological settings with asymmetric connectivity and the encoding for dynamical stimuli have not been well-characterized. Here, we study the collective response in a kinetic Ising model that encodes the dynamic input. We apply gradient-based method and mean-field approximation to reconstruct networks given the neural code that encodes dynamic input patterns. We measure network asymmetry, decoding performance, and entropy production from networks that generate optimal population code. We analyze how stimulus correlation, time scale, and reliability of the network affect optimal encoding networks. Specifically, we find network dynamics altered by statistics of the dynamic input, identify stimulus encoding strategies, and show optimal effective temperature in the asymmetric networks. We further discuss how this approach connects to the Bayesian framework and continuous recurrent neural networks. Together, these results bridge concepts of nonequilibrium physics with the analyses of dynamics and coding in networks.

Coarse-to-fine processing drives the efficient coding of natural scenes in mouse visual cortex

The visual system processes sensory inputs sequentially, perceiving coarse information before fine details. Here we study the neural basis of coarse-to-fine processing and its computational benefits in natural vision. We find that primary visual cortical neurons in awake mice respond to natural scenes in a coarse-to-fine manner, primarily driven by individual neurons rapidly shifting their spatial frequency preference from low to high over a brief response period. This shift transforms the population response in a way that counteracts the statistical regularities of natural scenes, thereby reducing redundancy and generating a more efficient neural representation. The increase in representational efficiency does not occur in either dark-reared or anesthetized mice, which show significantly attenuated coarse-to-fine spatial processing. Collectively, these results illustrate that coarse-to-fine processing is state dependent, develops postnatally via visual experience, and provides a computational advantage by generating more efficient representations of the complex spatial statistics of ethologically relevant natural scenes.

Skyberg et al. show that the visual cortex of mice processes natural scenes in a coarse-to-fine manner, driven by individual neuron’s temporal dynamics. These response dynamics, which require visual experience to develop, reduce redundancy in the neural code and lead to more efficient representations of complex visual stimuli.

Rat sensitivity to multipoint statistics is predicted by efficient coding of natural scenes

Efficient processing of sensory data requires adapting the neuronal encoding strategy to the statistics of natural stimuli. Previously, in Hermundstad et al., 2014, we showed that local multipoint correlation patterns that are most variable in natural images are also the most perceptually salient for human observers, in a way that is compatible with the efficient coding principle. Understanding the neuronal mechanisms underlying such adaptation to image statistics will require performing invasive experiments that are impossible in humans. Therefore, it is important to understand whether a similar phenomenon can be detected in animal species that allow for powerful experimental manipulations, such as rodents. Here we selected four image statistics (from single- to four-point correlations) and trained four groups of rats to discriminate between white noise patterns and binary textures containing variable intensity levels of one of such statistics. We interpreted the resulting psychometric data with an ideal observer model, finding a sharp decrease in sensitivity from two- to four-point correlations and a further decrease from four- to three-point. This ranking fully reproduces the trend we previously observed in humans, thus extending a direct demonstration of efficient coding to a species where neuronal and developmental processes can be interrogated and causally manipulated.

A precise and adaptive neural mechanism for predictive temporal processing in the frontal cortex

The theory of predictive processing posits that the brain computes expectations to process information predictively. Empirical evidence in support of this theory, however, is scarce and largely limited to sensory areas. Here, we report a precise and adaptive mechanism in the frontal cortex of non-human primates consistent with predictive processing of temporal events. We found that the speed of neural dynamics is precisely adjusted according to the average time of an expected stimulus. This speed adjustment, in turn, enables neurons to encode stimuli in terms of deviations from expectation. This lawful relationship was evident across multiple experiments and held true during learning: when temporal statistics underwent covert changes, neural responses underwent predictable changes that reflected the new mean. Together, these results highlight a precise mathematical relationship between temporal statistics in the environment and neural activity in the frontal cortex that may serve as a mechanism for predictive temporal processing.