Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system

Synapse-type-specific competitive Hebbian learning forms functional recurrent networks

Cortical networks exhibit complex stimulus-response patterns that are based on specific recurrent interactions between neurons. For example, the balance between excitatory and inhibitory currents has been identified as a central component of cortical computations. However, it remains unclear how the required synaptic connectivity can emerge in developing circuits where synapses between excitatory and inhibitory neurons are simultaneously plastic. Using theory and modeling, we propose that a wide range of cortical response properties can arise from a single plasticity paradigm that acts simultaneously at all excitatory and inhibitory connections-Hebbian learning that is stabilized by the synapse-type-specific competition for a limited supply of synaptic resources. In plastic recurrent circuits, this competition enables the formation and decorrelation of inhibition-balanced receptive fields. Networks develop an assembly structure with stronger synaptic connections between similarly tuned excitatory and inhibitory neurons and exhibit response normalization and orientation-specific center-surround suppression, reflecting the stimulus statistics during training. These results demonstrate how neurons can self-organize into functional networks and suggest an essential role for synapse-type-specific competitive learning in the development of cortical circuits.

Spiking networks that efficiently process dynamic sensory features explain receptor information mixing in somatosensory cortex

How do biological neural systems efficiently encode, transform and propagate information between the sensory periphery and the sensory cortex about sensory features evolving at different time scales? Are these computations efficient in normative information processing terms? While previous work has suggested that biologically plausible models of of such neural information processing may be implemented efficiently within a single processing layer, how such computations extend across several processing layers is less clear. Here, we model propagation of multiple time-varying sensory features across a sensory pathway, by extending the theory of efficient coding with spikes to efficient encoding, transformation and transmission of sensory signals. These computations are optimally realized by a multilayer spiking network with feedforward networks of spiking neurons (receptor layer) and recurrent excitatory-inhibitory networks of generalized leaky integrate-and-fire neurons (recurrent layers). Our model efficiently realizes a broad class of feature transformations, including positive and negative interaction across features, through specific and biologically plausible structures of feedforward connectivity. We find that mixing of sensory features in the activity of single neurons is beneficial because it lowers the metabolic cost at the network level. We apply the model to the somatosensory pathway by constraining it with parameters measured empirically and include in its last node, analogous to the primary somatosensory cortex (S1), two types of inhibitory neurons: parvalbumin-positive neurons realizing lateral inhibition, and somatostatin-positive neurons realizing winner-take-all inhibition. By implementing a negative interaction across stimulus features, this model captures several intriguing empirical observations from the somatosensory system of the mouse, including a decrease of sustained responses from subcortical networks to S1, a non-linear effect of the knock-out of receptor neuron types on the activity in S1, and amplification of weak signals from sensory neurons across the pathway.

Positive Competitive Networks for Sparse Reconstruction

We propose and analyze a continuous-time firing-rate neural network, the positive firing-rate competitive network (PFCN), to tackle sparse reconstruction problems with non-negativity constraints. These problems, which involve approximating a given input stimulus from a dictionary using a set of sparse (active) neurons, play a key role in a wide range of domains, including, for example, neuroscience, signal processing, and machine learning. First, by leveraging the theory of proximal operators, we relate the equilibria of a family of continuous-time firing-rate neural networks to the optimal solutions of sparse reconstruction problems. Then we prove that the PFCN is a positive system and give rigorous conditions for the convergence to the equilibrium. Specifically, we show that the convergence depends only on a property of the dictionary and is linear-exponential in the sense that initially, the convergence rate is at worst linear and then, after a transient, becomes exponential. We also prove a number of technical results to assess the contractivity properties of the neural dynamics of interest. Our analysis leverages contraction theory to characterize the behavior of a family of firing-rate competitive networks for sparse reconstruction with and without non-negativity constraints. Finally, we validate the effectiveness of our approach via a numerical example.

Structure, dynamics, coding and optimal biophysical parameters of efficient excitatory-inhibitory spiking networks

The principle of efficient coding posits that sensory cortical networks are designed to encode maximal sensory information with minimal metabolic cost. Despite the major influence of efficient coding in neuro-science, it has remained unclear whether fundamental empirical properties of neural network activity can be explained solely based on this normative principle. Here, we rigorously derive the structural, coding, biophysical and dynamical properties of excitatory-inhibitory recurrent networks of spiking neurons that emerge directly from imposing that the network minimizes an instantaneous loss function and a time-averaged performance measure enacting efficient coding. The optimal network has biologically-plausible biophysical features, including realistic integrate-and-fire spiking dynamics, spike-triggered adaptation, and a non-stimulus-specific excitatory external input regulating metabolic cost. The efficient network has excitatory-inhibitory recurrent connectivity between neurons with similar stimulus tuning implementing feature-specific competition, similar to that recently found in visual cortex. Networks with unstructured connectivity cannot reach comparable levels of coding efficiency. The optimal biophysical parameters include 4 to 1 ratio of excitatory vs inhibitory neurons and 3 to 1 ratio of mean inhibitory-to-inhibitory vs. excitatory-to-inhibitory connectivity that closely match those of cortical sensory networks. The efficient network has biologically-plausible spiking dynamics, with a tight instantaneous E-I balance that makes them capable to achieve efficient coding of external stimuli varying over multiple time scales. Together, these results explain how efficient coding may be implemented in cortical networks and suggests that key properties of biological neural networks may be accounted for by efficient coding.

Specific connectivity optimizes learning in thalamocortical loops

Thalamocortical loops have a central role in cognition and motor control, but precisely how they contribute to these processes is unclear. Recent studies showing evidence of plasticity in thalamocortical synapses indicate a role for the thalamus in shaping cortical dynamics through learning. Since signals undergo a compression from the cortex to the thalamus, we hypothesized that the computational role of the thalamus depends critically on the structure of corticothalamic connectivity. To test this, we identified the optimal corticothalamic structure that promotes biologically plausible learning in thalamocortical synapses. We found that corticothalamic projections specialized to communicate an efference copy of the cortical output benefit motor control, while communicating the modes of highest variance is optimal for working memory tasks. We analyzed neural recordings from mice performing grasping and delayed discrimination tasks and found corticothalamic communication consistent with these predictions. These results suggest that the thalamus orchestrates cortical dynamics in a functionally precise manner through structured connectivity.

Generalizing biological surround suppression based on center surround similarity via deep neural network models

Sensory perception is dramatically influenced by the context. Models of contextual neural surround effects in vision have mostly accounted for Primary Visual Cortex (V1) data, via nonlinear computations such as divisive normalization. However, surround effects are not well understood within a hierarchy, for neurons with more complex stimulus selectivity beyond V1. We utilized feedforward deep convolutional neural networks and developed a gradient-based technique to visualize the most suppressive and excitatory surround. We found that deep neural networks exhibited a key signature of surround effects in V1, highlighting center stimuli that visually stand out from the surround and suppressing responses when the surround stimulus is similar to the center. We found that in some neurons, especially in late layers, when the center stimulus was altered, the most suppressive surround surprisingly can follow the change. Through the visualization approach, we generalized previous understanding of surround effects to more complex stimuli, in ways that have not been revealed in visual cortices. In contrast, the suppression based on center surround similarity was not observed in an untrained network. We identified further successes and mismatches of the feedforward CNNs to the biology. Our results provide a testable hypothesis of surround effects in higher visual cortices, and the visualization approach could be adopted in future biological experimental designs.

Computational methods to study information processing in neural circuits

The brain is an information processing machine and thus naturally lends itself to be studied using computational tools based on the principles of information theory. For this reason, computational methods based on or inspired by information theory have been a cornerstone of practical and conceptual progress in neuroscience. In this Review, we address how concepts and computational tools related to information theory are spurring the development of principled theories of information processing in neural circuits and the development of influential mathematical methods for the analyses of neural population recordings. We review how these computational approaches reveal mechanisms of essential functions performed by neural circuits. These functions include efficiently encoding sensory information and facilitating the transmission of information to downstream brain areas to inform and guide behavior. Finally, we discuss how further progress and insights can be achieved, in particular by studying how competing requirements of neural encoding and readout may be optimally traded off to optimize neural information processing.

Object Boundary Detection in Natural Images May Depend on "Incitatory" Cell-Cell Interactions

Detecting object boundaries is crucial for recognition, but how the process unfolds in visual cortex remains unknown. To study the problem faced by a hypothetical boundary cell, and to predict how cortical circuitry could produce a boundary cell from a population of conventional "simple cells," we labeled 30,000 natural image patches and used Bayes' rule to help determine how a simple cell should influence a nearby boundary cell depending on its relative offset in receptive field position and orientation. We identified the following three basic types of cell-cell interactions: rising and falling interactions with a range of slopes and saturation rates, and nonmonotonic (bump-shaped) interactions with varying modes and amplitudes. Using simple models, we show that a ubiquitous cortical circuit motif consisting of direct excitation and indirect inhibition-a compound effect we call "incitation"-can produce the entire spectrum of simple cell-boundary cell interactions found in our dataset. Moreover, we show that the synaptic weights that parameterize an incitation circuit can be learned by a single-layer "delta" rule. We conclude that incitatory interconnections are a generally useful computing mechanism that the cortex may exploit to help solve difficult natural classification problems.SIGNIFICANCE STATEMENT Simple cells in primary visual cortex (V1) respond to oriented edges and have long been supposed to detect object boundaries, yet the prevailing model of a simple cell-a divisively normalized linear filter-is a surprisingly poor natural boundary detector. To understand why, we analyzed image statistics on and off object boundaries, allowing us to characterize the neural-style computations needed to perform well at this difficult natural classification task. We show that a simple circuit motif known to exist in V1 is capable of extracting high-quality boundary probability signals from local populations of simple cells. Our findings suggest a new, more general way of conceptualizing cell-cell interconnections in the cortex.

Intermodulation from Unisensory to Multisensory Perception: A Review

Previous intermodulation (IM) studies have employed two (or more) temporal modulations of a stimulus, with different local elements of the stimulus being modulated by different frequencies. Brain activities of IM obtained mainly from electroencephalograms (EEG) have been analyzed in the frequency domain. As a powerful tool, IM, which can provide a direct and objective physiological measure of neural interaction, has emerged as a promising method to decipher neural interactions in visual perception, and reveal the underlying different perceptual processing levels. In this review, we summarize the recent applications of IM in visual perception, detail the protocols and types of IM, and extend its utility and potential applications to the multisensory domain. We propose that using IM could prevail in partially revealing the potential hierarchical processing of multisensory information and contribute to a deeper understanding of the underlying brain dynamics.

Visually sensitive seizures: An updated review by the Epilepsy Foundation

Light flashes, patterns, or color changes can provoke seizures in up to 1 in 4000 persons. Prevalence may be higher because of selection bias. The Epilepsy Foundation reviewed light-induced seizures in 2005. Since then, images on social media, virtual reality, three-dimensional (3D) movies, and the Internet have proliferated. Hundreds of studies have explored the mechanisms and presentations of photosensitive seizures, justifying an updated review. This literature summary derives from a nonsystematic literature review via PubMed using the terms "photosensitive" and "epilepsy." The photoparoxysmal response (PPR) is an electroencephalography (EEG) phenomenon, and photosensitive seizures (PS) are seizures provoked by visual stimulation. Photosensitivity is more common in the young and in specific forms of generalized epilepsy. PS can coexist with spontaneous seizures. PS are hereditable and linked to recently identified genes. Brain imaging usually is normal, but special studies imaging white matter tracts demonstrate abnormal connectivity. Occipital cortex and connected regions are hyperexcitable in subjects with light-provoked seizures. Mechanisms remain unclear. Video games, social media clips, occasional movies, and natural stimuli can provoke PS. Virtual reality and 3D images so far appear benign unless they contain specific provocative content, for example, flashes. Images with flashes brighter than 20 candelas/m² at 3-60 (particularly 15-20) Hz occupying at least 10 to 25% of the visual field are a risk, as are red color flashes or oscillating stripes. Equipment to assay for these characteristics is probably underutilized. Prevention of seizures includes avoiding provocative stimuli, covering one eye, wearing dark glasses, sitting at least two meters from screens, reducing contrast, and taking certain antiseizure drugs. Measurement of PPR suppression in a photosensitivity model can screen putative antiseizure drugs. Some countries regulate media to reduce risk. Visually-induced seizures remain significant public health hazards so they warrant ongoing scientific and regulatory efforts and public education.

Constrained brain volume in an efficient coding model explains the fraction of excitatory and inhibitory neurons in sensory cortices

The number of neurons in mammalian cortex varies by multiple orders of magnitude across different species. In contrast, the ratio of excitatory to inhibitory neurons (E:I ratio) varies in a much smaller range, from 3:1 to 9:1 and remains roughly constant for different sensory areas within a species. Despite this structure being important for understanding the function of neural circuits, the reason for this consistency is not yet understood. While recent models of vision based on the efficient coding hypothesis show that increasing the number of both excitatory and inhibitory cells improves stimulus representation, the two cannot increase simultaneously due to constraints on brain volume. In this work, we implement an efficient coding model of vision under a constraint on the volume (using number of neurons as a surrogate) while varying the E:I ratio. We show that the performance of the model is optimal at biologically observed E:I ratios under several metrics. We argue that this happens due to trade-offs between the computational accuracy and the representation capacity for natural stimuli. Further, we make experimentally testable predictions that 1) the optimal E:I ratio should be higher for species with a higher sparsity in the neural activity and 2) the character of inhibitory synaptic distributions and firing rates should change depending on E:I ratio. Our findings, which are supported by our new preliminary analyses of publicly available data, provide the first quantitative and testable hypothesis based on optimal coding models for the distribution of excitatory and inhibitory neural types in the mammalian sensory cortices.

Neurons in the brain come in two main types: excitatory and inhibitory. The interplay between them shapes neural computation. Despite brain sizes varying by several orders of magnitude across species, the ratio of excitatory and inhibitory sub-populations (E:I ratio) remains relatively constant, and we don’t know why. Simulations of theoretical models of the brain can help answer such questions, especially when experiments are prohibitive or impossible. Here we placed one such theoretical model of sensory coding (’sparse coding’ that minimizes the simultaneously active neurons) under a biophysical ‘volume’ constraint that fixes the total number of neurons available. We vary the E:I ratio in the model (which cannot be done in experiments), and reveal an optimal E:I ratio where the representation of sensory stimulus and energy consumption within the circuit are concurrently optimal. We also show that varying the population sparsity changes the optimal E:I ratio, spanning the relatively narrow ranges observed in biology. Crucially, this minimally parameterized theoretical model makes predictions about structure (recurrent connectivity) and activity (population sparsity) in neural circuits with different E:I ratios (i.e., different species), of which we verify the latter in a first-of-its-kind inter-species comparison using newly publicly available data.

Learning an Efficient Hippocampal Place Map from Entorhinal Inputs Using Non-Negative Sparse Coding

Cells in the entorhinal cortex (EC) contain rich spatial information and project strongly to the hippocampus where a cognitive map is supposedly created. These cells range from cells with structured spatial selectivity, such as grid cells in the medial EC (MEC) that are selective to an array of spatial locations that form a hexagonal grid, to weakly spatial cells, such as non-grid cells in the MEC and lateral EC (LEC) that contain spatial information but have no structured spatial selectivity. However, in a small environment, place cells in the hippocampus are generally selective to a single location of the environment, while granule cells in the dentate gyrus of the hippocampus have multiple discrete firing locations but lack spatial periodicity. Given the anatomic connection from the EC to the hippocampus, how the hippocampus retrieves information from upstream EC remains unclear. Here, we propose a unified learning model that can describe the spatial tuning properties of both hippocampal place cells and dentate gyrus granule cells based on non-negative sparse coding from EC inputs. Sparse coding plays an important role in many cortical areas and is proposed here to have a key role in the hippocampus. Our results show that the hexagonal patterns of MEC grid cells with various orientations, grid spacings and phases are necessary for the model to learn different place cells that efficiently tile the entire spatial environment. However, if there is a lack of diversity in any grid parameters or a lack of hippocampal cells in the network, this will lead to the emergence of hippocampal cells that have multiple firing locations. More surprisingly, the model can also learn hippocampal place cells even when weakly spatial cells, instead of grid cells, are used as the input to the hippocampus. This work suggests that sparse coding may be one of the underlying organizing principles for the navigational system of the brain.

Neuronal variability reflects probabilistic inference tuned to natural image statistics

Neuronal activity in sensory cortex fluctuates over time and across repetitions of the same input. This variability is often considered detrimental to neural coding. The theory of neural sampling proposes instead that variability encodes the uncertainty of perceptual inferences. In primary visual cortex (V1), modulation of variability by sensory and non-sensory factors supports this view. However, it is unknown whether V1 variability reflects the statistical structure of visual inputs, as would be required for inferences correctly tuned to the statistics of the natural environment. Here we combine analysis of image statistics and recordings in macaque V1 to show that probabilistic inference tuned to natural image statistics explains the widely observed dependence between spike count variance and mean, and the modulation of V1 activity and variability by spatial context in images. Our results show that the properties of a basic aspect of cortical responses-their variability-can be explained by a probabilistic representation tuned to naturalistic inputs.

Learning divisive normalization in primary visual cortex

Divisive normalization (DN) is a prominent computational building block in the brain that has been proposed as a canonical cortical operation. Numerous experimental studies have verified its importance for capturing nonlinear neural response properties to simple, artificial stimuli, and computational studies suggest that DN is also an important component for processing natural stimuli. However, we lack quantitative models of DN that are directly informed by measurements of spiking responses in the brain and applicable to arbitrary stimuli. Here, we propose a DN model that is applicable to arbitrary input images. We test its ability to predict how neurons in macaque primary visual cortex (V1) respond to natural images, with a focus on nonlinear response properties within the classical receptive field. Our model consists of one layer of subunits followed by learned orientation-specific DN. It outperforms linear-nonlinear and wavelet-based feature representations and makes a significant step towards the performance of state-of-the-art convolutional neural network (CNN) models. Unlike deep CNNs, our compact DN model offers a direct interpretation of the nature of normalization. By inspecting the learned normalization pool of our model, we gained insights into a long-standing question about the tuning properties of DN that update the current textbook description: we found that within the receptive field oriented features were normalized preferentially by features with similar orientation rather than non-specifically as currently assumed.

Improved inference in coupling, encoding, and decoding models and its consequence for neuroscientific interpretation

BACKGROUND

A central goal of systems neuroscience is to understand the relationships amongst constituent units in neural populations, and their modulation by external factors, using high-dimensional and stochastic neural recordings. Parametric statistical models (e.g., coupling, encoding, and decoding models), play an instrumental role in accomplishing this goal. However, extracting conclusions from a parametric model requires that it is fit using an inference algorithm capable of selecting the correct parameters and properly estimating their values. Traditional approaches to parameter inference have been shown to suffer from failures in both selection and estimation. The recent development of algorithms that ameliorate these deficiencies raises the question of whether past work relying on such inference procedures have produced inaccurate systems neuroscience models, thereby impairing their interpretation.

NEW METHOD

We used algorithms based on Union of Intersections, a statistical inference framework based on stability principles, capable of improved selection and estimation.

COMPARISON

We fit functional coupling, encoding, and decoding models across a battery of neural datasets using both UoI and baseline inference procedures (e.g., ℓ₁-penalized GLMs), and compared the structure of their fitted parameters.

RESULTS

Across recording modality, brain region, and task, we found that UoI inferred models with increased sparsity, improved stability, and qualitatively different parameter distributions, while maintaining predictive performance. We obtained highly sparse functional coupling networks with substantially different community structure, more parsimonious encoding models, and decoding models that relied on fewer single-units.

CONCLUSIONS

Together, these results demonstrate that improved parameter inference, achieved via UoI, reshapes interpretation in diverse neuroscience contexts.

Learning receptive field properties of complex cells in V1

There are two distinct classes of cells in the primary visual cortex (V1): simple cells and complex cells. One defining feature of complex cells is their spatial phase invariance; they respond strongly to oriented grating stimuli with a preferred orientation but with a wide range of spatial phases. A classical model of complete spatial phase invariance in complex cells is the energy model, in which the responses are the sum of the squared outputs of two linear spatially phase-shifted filters. However, recent experimental studies have shown that complex cells have a diverse range of spatial phase invariance and only a subset can be characterized by the energy model. While several models have been proposed to explain how complex cells could learn to be selective to orientation but invariant to spatial phase, most existing models overlook many biologically important details. We propose a biologically plausible model for complex cells that learns to pool inputs from simple cells based on the presentation of natural scene stimuli. The model is a three-layer network with rate-based neurons that describes the activities of LGN cells (layer 1), V1 simple cells (layer 2), and V1 complex cells (layer 3). The first two layers implement a recently proposed simple cell model that is biologically plausible and accounts for many experimental phenomena. The neural dynamics of the complex cells is modeled as the integration of simple cells inputs along with response normalization. Connections between LGN and simple cells are learned using Hebbian and anti-Hebbian plasticity. Connections between simple and complex cells are learned using a modified version of the Bienenstock, Cooper, and Munro (BCM) rule. Our results demonstrate that the learning rule can describe a diversity of complex cells, similar to those observed experimentally.

Many cortical functions originate from the learning ability of the brain. How the properties of cortical cells are learned is vital for understanding how the brain works. There are many models that explain how V1 simple cells can be learned. However, how V1 complex cells are learned still remains unclear. In this paper, we propose a model of learning in complex cells based on the Bienenstock, Cooper, and Munro (BCM) rule. We demonstrate that properties of receptive fields of complex cells can be learned using this biologically plausible learning rule. Quantitative comparisons between the model and experimental data are performed. Results show that model complex cells can account for the diversity of complex cells found in experimental studies. In summary, this study provides a plausible explanation for how complex cells can be learned using biologically plausible plasticity mechanisms. Our findings help us to better understand biological vision processing and provide us with insights into the general signal processing principles that the visual cortex employs to process visual information.

Selectivity and robustness of sparse coding networks

We investigate how the population nonlinearities resulting from lateral inhibition and thresholding in sparse coding networks influence neural response selectivity and robustness. We show that when compared to pointwise nonlinear models, such population nonlinearities improve the selectivity to a preferred stimulus and protect against adversarial perturbations of the input. These findings are predicted from the geometry of the single-neuron iso-response surface, which provides new insight into the relationship between selectivity and adversarial robustness. Inhibitory lateral connections curve the iso-response surface outward in the direction of selectivity. Since adversarial perturbations are orthogonal to the iso-response surface, adversarial attacks tend to be aligned with directions of selectivity. Consequently, the network is less easily fooled by perceptually irrelevant perturbations to the input. Together, these findings point to benefits of integrating computational principles found in biological vision systems into artificial neural networks.

Contextual Integration in Cortical and Convolutional Neural Networks

It has been suggested that neurons can represent sensory input using probability distributions and neural circuits can perform probabilistic inference. Lateral connections between neurons have been shown to have non-random connectivity and modulate responses to stimuli within the classical receptive field. Large-scale efforts mapping local cortical connectivity describe cell type specific connections from inhibitory neurons and like-to-like connectivity between excitatory neurons. To relate the observed connectivity to computations, we propose a neuronal network model that approximates Bayesian inference of the probability of different features being present at different image locations. We show that the lateral connections between excitatory neurons in a circuit implementing contextual integration in this should depend on correlations between unit activities, minus a global inhibitory drive. The model naturally suggests the need for two types of inhibitory gates (normalization, surround inhibition). First, using natural scene statistics and classical receptive fields corresponding to simple cells parameterized with data from mouse primary visual cortex, we show that the predicted connectivity qualitatively matches with that measured in mouse cortex: neurons with similar orientation tuning have stronger connectivity, and both excitatory and inhibitory connectivity have a modest spatial extent, comparable to that observed in mouse visual cortex. We incorporate lateral connections learned using this model into convolutional neural networks. Features are defined by supervised learning on the task, and the lateral connections provide an unsupervised learning of feature context in multiple layers. Since the lateral connections provide contextual information when the feedforward input is locally corrupted, we show that incorporating such lateral connections into convolutional neural networks makes them more robust to noise and leads to better performance on noisy versions of the MNIST dataset. Decomposing the predicted lateral connectivity matrices into low-rank and sparse components introduces additional cell types into these networks. We explore effects of cell-type specific perturbations on network computation. Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding computations implemented by feedforward connections when the input is unreliable and demonstrate the potential usefulness of combining supervised and unsupervised learning techniques in real-world vision tasks.

Spatial suppression in visual motion perception is driven by inhibition: Evidence from MEG gamma oscillations

Spatial suppression (SS) is a visual perceptual phenomenon that is manifest in a reduction of directional sensitivity for drifting high-contrast gratings whose size exceeds the center of the visual field. Gratings moving at faster velocities induce stronger SS. The neural processes that give rise to such size- and velocity-dependent reductions in directional sensitivity are currently unknown, and the role of surround inhibition is unclear. In magnetoencephalogram (MEG), large high-contrast drifting gratings induce a strong gamma response (GR), which also attenuates with an increase in the gratings' velocity. It has been suggested that the slope of this GR attenuation is mediated by inhibitory interactions in the primary visual cortex. Herein, we investigate whether SS is related to this inhibitory-based MEG measure. We evaluated SS and GR in two independent samples of participants: school-age boys and adult women. The slope of GR attenuation predicted inter-individual differences in SS in both samples. Test-retest reliability of the neuro-behavioral correlation was assessed in the adults, and was high between two sessions separated by several days or weeks. Neither frequencies nor absolute amplitudes of the GRs correlated with SS, which highlights the functional relevance of velocity-related changes in GR magnitude caused by augmentation of incoming input. Our findings provide evidence that links the psychophysical phenomenon of SS to inhibitory-based neural responses in the human primary visual cortex. This supports the role of inhibitory interactions as an important underlying mechanism for spatial suppression.

Constrained inference in sparse coding reproduces contextual effects and predicts laminar neural dynamics

When probed with complex stimuli that extend beyond their classical receptive field, neurons in primary visual cortex display complex and non-linear response characteristics. Sparse coding models reproduce some of the observed contextual effects, but still fail to provide a satisfactory explanation in terms of realistic neural structures and cortical mechanisms, since the connection scheme they propose consists only of interactions among neurons with overlapping input fields. Here we propose an extended generative model for visual scenes that includes spatial dependencies among different features. We derive a neurophysiologically realistic inference scheme under the constraint that neurons have direct access only to local image information. The scheme can be interpreted as a network in primary visual cortex where two neural populations are organized in different layers within orientation hypercolumns that are connected by local, short-range and long-range recurrent interactions. When trained with natural images, the model predicts a connectivity structure linking neurons with similar orientation preferences matching the typical patterns found for long-ranging horizontal axons and feedback projections in visual cortex. Subjected to contextual stimuli typically used in empirical studies, our model replicates several hallmark effects of contextual processing and predicts characteristic differences for surround modulation between the two model populations. In summary, our model provides a novel framework for contextual processing in the visual system proposing a well-defined functional role for horizontal axons and feedback projections.

Integrating Flexible Normalization into Midlevel Representations of Deep Convolutional Neural Networks

Deep convolutional neural networks (CNNs) are becoming increasingly popular models to predict neural responses in visual cortex. However, contextual effects, which are prevalent in neural processing and in perception, are not explicitly handled by current CNNs, including those used for neural prediction. In primary visual cortex, neural responses are modulated by stimuli spatially surrounding the classical receptive field in rich ways. These effects have been modeled with divisive normalization approaches, including flexible models, where spatial normalization is recruited only to the degree that responses from center and surround locations are deemed statistically dependent. We propose a flexible normalization model applied to midlevel representations of deep CNNs as a tractable way to study contextual normalization mechanisms in midlevel cortical areas. This approach captures nontrivial spatial dependencies among midlevel features in CNNs, such as those present in textures and other visual stimuli, that arise from tiling high-order features geometrically. We expect that the proposed approach can make predictions about when spatial normalization might be recruited in midlevel cortical areas. We also expect this approach to be useful as part of the CNN tool kit, therefore going beyond more restrictive fixed forms of normalization.

Learning Nonclassical Receptive Field Modulation for Contour Detection

This work develops a biologically inspired neural network for contour detection in natural images by combining the nonclassical receptive field modulation mechanism with a deep learning framework. The input image is first convolved with the local feature detectors to produce the classical receptive field responses, and then a corresponding modulatory kernel is constructed for each feature map to model the nonclassical receptive field modulation behaviors. The modulatory effects can activate a larger cortical area and thus allow cortical neurons to integrate a broader range of visual information to recognize complex cases. Additionally, to characterize spatial structures at various scales, a multiresolution technique is used to represent visual field information from fine to coarse. Different scale responses are combined to estimate the contour probability. Our method achieves state-of-the-art results among all biologically inspired contour detection models. This study provides a method for improving visual modeling of contour detection and inspires new ideas for integrating more brain cognitive mechanisms into deep neural networks.

On the Sparse Structure of Natural Sounds and Natural Images: Similarities, Differences, and Implications for Neural Coding

Sparse coding models of natural images and sounds have been able to predict several response properties of neurons in the visual and auditory systems. While the success of these models suggests that the structure they capture is universal across domains to some degree, it is not yet clear which aspects of this structure are universal and which vary across sensory modalities. To address this, we fit complete and highly overcomplete sparse coding models to natural images and spectrograms of speech and report on differences in the statistics learned by these models. We find several types of sparse features in natural images, which all appear in similar, approximately Laplace distributions, whereas the many types of sparse features in speech exhibit a broad range of sparse distributions, many of which are highly asymmetric. Moreover, individual sparse coding units tend to exhibit higher lifetime sparseness for overcomplete models trained on images compared to those trained on speech. Conversely, population sparseness tends to be greater for these networks trained on speech compared with sparse coding models of natural images. To illustrate the relevance of these findings to neural coding, we studied how they impact a biologically plausible sparse coding network's representations in each sensory modality. In particular, a sparse coding network with synaptically local plasticity rules learns different sparse features from speech data than are found by more conventional sparse coding algorithms, but the learned features are qualitatively the same for these models when trained on natural images.

From intermodulation components to visual perception and cognition-a review

Perception results from complex interactions among sensory and cognitive processes across hierarchical levels in the brain. Intermodulation (IM) components, used in frequency tagging neuroimaging designs, have emerged as a promising direct measure of such neural interactions. IMs have initially been used in electroencephalography (EEG) to investigate low-level visual processing. In a more recent trend, IMs in EEG and other neuroimaging methods are being used to shed light on mechanisms of mid- and high-level perceptual processes, including the involvement of cognitive functions such as attention and expectation. Here, we provide an account of various mechanisms that may give rise to IMs in neuroimaging data, and what these IMs may look like. We discuss methodologies that can be implemented for different uses of IMs and we demonstrate how IMs can provide insights into the existence, the degree and the type of neural integration mechanisms at hand. We then review a range of recent studies exploiting IMs in visual perception research, placing an emphasis on high-level vision and the influence of awareness and cognition on visual processing. We conclude by suggesting future directions that can enhance the benefits of IM-methodology in perception research.

Toward a Biologically Plausible Model of LGN-V1 Pathways Based on Efficient Coding

Increasing evidence supports the hypothesis that the visual system employs a sparse code to represent visual stimuli, where information is encoded in an efficient way by a small population of cells that respond to sensory input at a given time. This includes simple cells in primary visual cortex (V1), which are defined by their linear spatial integration of visual stimuli. Various models of sparse coding have been proposed to explain physiological phenomena observed in simple cells. However, these models have usually made the simplifying assumption that inputs to simple cells already incorporate linear spatial summation. This overlooks the fact that these inputs are known to have strong non-linearities such the separation of ON and OFF pathways, or separation of excitatory and inhibitory neurons. Consequently these models ignore a range of important experimental phenomena that are related to the emergence of linear spatial summation from non-linear inputs, such as segregation of ON and OFF sub-regions of simple cell receptive fields, the push-pull effect of excitation and inhibition, and phase-reversed cortico-thalamic feedback. Here, we demonstrate that a two-layer model of the visual pathway from the lateral geniculate nucleus to V1 that incorporates these biological constraints on the neural circuits and is based on sparse coding can account for the emergence of these experimental phenomena, diverse shapes of receptive fields and contrast invariance of orientation tuning of simple cells when the model is trained on natural images. The model suggests that sparse coding can be implemented by the V1 simple cells using neural circuits with a simple biologically plausible architecture.

Normalization and pooling in hierarchical models of natural images

Divisive normalization and subunit pooling are two canonical classes of computation that have become widely used in descriptive (what) models of visual cortical processing. Normative (why) models from natural image statistics can help constrain the form and parameters of such classes of models. We focus on recent advances in two particular directions, namely deriving richer forms of divisive normalization, and advances in learning pooling from image statistics. We discuss the incorporation of such components into hierarchical models. We consider both hierarchical unsupervised learning from image statistics, and discriminative supervised learning in deep convolutional neural networks (CNNs). We further discuss studies on the utility and extensions of the convolutional architecture, which has also been adopted by recent descriptive models. We review the recent literature and discuss the current promises and gaps of using such approaches to gain a better understanding of how cortical neurons represent and process complex visual stimuli.

Surface color and predictability determine contextual modulation of V1 firing and gamma oscillations

The integration of direct bottom-up inputs with contextual information is a core feature of neocortical circuits. In area V1, neurons may reduce their firing rates when their receptive field input can be predicted by spatial context. Gamma-synchronized (30–80 Hz) firing may provide a complementary signal to rates, reflecting stronger synchronization between neuronal populations receiving mutually predictable inputs. We show that large uniform surfaces, which have high spatial predictability, strongly suppressed firing yet induced prominent gamma synchronization in macaque V1, particularly when they were colored. Yet, chromatic mismatches between center and surround, breaking predictability, strongly reduced gamma synchronization while increasing firing rates. Differences between responses to different colors, including strong gamma-responses to red, arose from stimulus adaptation to a full-screen background, suggesting prominent differences in adaptation between M- and L-cone signaling pathways. Thus, synchrony signaled whether RF inputs were predicted from spatial context, while firing rates increased when stimuli were unpredicted from context.

Stimulus- and goal-oriented frameworks for understanding natural vision

Our knowledge of sensory processing has advanced dramatically in the last few decades, but this understanding remains far from complete, especially for stimuli with the large dynamic range and strong temporal and spatial correlations characteristic of natural visual inputs. Here we describe some of the issues that make understanding the encoding of natural images a challenge. We highlight two broad strategies for approaching this problem: a stimulus-oriented framework and a goal-oriented one. Different contexts can call for one framework or the other. Looking forward, recent advances, particularly those based in machine learning, show promise in borrowing key strengths of both frameworks and by doing so illuminating a path to a more comprehensive understanding of the encoding of natural stimuli.

Adaptive scales of integration and response latencies in a critically-balanced model of the primary visual cortex

The primary visual cortex (V1) integrates information over scales in visual space, which have been shown to vary, in an input-dependent manner, as a function of contrast and other visual parameters. Which algorithms the brain uses to achieve this feat are largely unknown and an open problem in visual neuroscience. We demonstrate that a simple dynamical mechanism can account for this contrast-dependent scale of integration in visuotopic space as well as connect this property to two other stimulus-dependent features of V1: extents of lateral integration on the cortical surface and response latencies.

Learning Midlevel Auditory Codes from Natural Sound Statistics

Interaction with the world requires an organism to transform sensory signals into representations in which behaviorally meaningful properties of the environment are made explicit. These representations are derived through cascades of neuronal processing stages in which neurons at each stage recode the output of preceding stages. Explanations of sensory coding may thus involve understanding how low-level patterns are combined into more complex structures. To gain insight into such midlevel representations for sound, we designed a hierarchical generative model of natural sounds that learns combinations of spectrotemporal features from natural stimulus statistics. In the first layer, the model forms a sparse convolutional code of spectrograms using a dictionary of learned spectrotemporal kernels. To generalize from specific kernel activation patterns, the second layer encodes patterns of time-varying magnitude of multiple first-layer coefficients. When trained on corpora of speech and environmental sounds, some second-layer units learned to group similar spectrotemporal features. Others instantiate opponency between distinct sets of features. Such groupings might be instantiated by neurons in the auditory cortex, providing a hypothesis for midlevel neuronal computation.

Neurophysiological Analysis of the Genesis Mechanism of EEG During the Interictal and Ictal Periods Using a Multiple Neural Masses Model

Electroencephalography (EEG) is an important method to investigate the neurophysiological mechanism underlying epileptogenesis to identify new therapies for the treatment of epilepsy. The neurophysiologically based neural mass model (NMM) can build a bridge between signal processing and neurophysiology, which can be used as a platform to explore the neurophysiological mechanism of epileptogenesis. Most EEG signals cannot be regarded as the outputs of a single NMM with identical model parameters. The outputs of NMM are simple because the diversity of neural signals in the same NMM is ignored. To improve the simulation of EEG signals, a multiple NMM is proposed, the output of which is the linear combination of the outputs of all NMMs. The NMM number is not fixed and is minimized under the premise of guaranteeing the fitting effect. Orthogonal matching pursuit is used to solve a constrained [Formula: see text] norm minimization problem for NMM number and the strength of every NMM. The results showed that the NMM number was significantly lower during the ictal period than during the interictal period, and the strength of major NMMs increased. This indicates that neural masses fuse into fewer larger neural masses with greater strength. The distribution of excitatory and inhibitory strength during the ictal and interictal periods was similar, whereas the excitation/inhibition ratio was higher during the ictal period than during the interictal period.

Contextual Interactions in Grating Plaid Configurations Are Explained by Natural Image Statistics and Neural Modeling

Processing natural scenes requires the visual system to integrate local features into global object descriptions. To achieve coherent representations, the human brain uses statistical dependencies to guide weighting of local feature conjunctions. Pairwise interactions among feature detectors in early visual areas may form the early substrate of these local feature bindings. To investigate local interaction structures in visual cortex, we combined psychophysical experiments with computational modeling and natural scene analysis. We first measured contrast thresholds for 2 × 2 grating patch arrangements (plaids), which differed in spatial frequency composition (low, high, or mixed), number of grating patch co-alignments (0, 1, or 2), and inter-patch distances (1° and 2° of visual angle). Contrast thresholds for the different configurations were compared to the prediction of probability summation (PS) among detector families tuned to the four retinal positions. For 1° distance the thresholds for all configurations were larger than predicted by PS, indicating inhibitory interactions. For 2° distance, thresholds were significantly lower compared to PS when the plaids were homogeneous in spatial frequency and orientation, but not when spatial frequencies were mixed or there was at least one misalignment. Next, we constructed a neural population model with horizontal laminar structure, which reproduced the detection thresholds after adaptation of connection weights. Consistent with prior work, contextual interactions were medium-range inhibition and long-range, orientation-specific excitation. However, inclusion of orientation-specific, inhibitory interactions between populations with different spatial frequency preferences were crucial for explaining detection thresholds. Finally, for all plaid configurations we computed their likelihood of occurrence in natural images. The likelihoods turned out to be inversely related to the detection thresholds obtained at larger inter-patch distances. However, likelihoods were almost independent of inter-patch distance, implying that natural image statistics could not explain the crowding-like results at short distances. This failure of natural image statistics to resolve the patch distance modulation of plaid visibility remains a challenge to the approach.

Early suppression effect in human primary visual cortex during Kanizsa illusion processing: A magnetoencephalographic evidence

Detection of illusory contours (ICs) such as Kanizsa figures is known to depend primarily upon the lateral occipital complex. Yet there is no universal agreement on the role of the primary visual cortex in this process; some existing evidence hints that an early stage of the visual response in V1 may involve relative suppression to Kanizsa figures compared with controls. Iso-oriented luminance borders, which are responsible for Kanizsa illusion, may evoke surround suppression in V1 and adjacent areas leading to the reduction in the initial response to Kanizsa figures. We attempted to test the existence, as well as to find localization and timing of the early suppression effect produced by Kanizsa figures in adult nonclinical human participants. We used two sizes of visual stimuli (4.5 and 9.0°) in order to probe the effect at two different levels of eccentricity; the stimuli were presented centrally in passive viewing conditions. We recorded magnetoencephalogram, which is more sensitive than electroencephalogram to activity originating from V1 and V2 areas. We restricted our analysis to the medial occipital area and the occipital pole, and to a 40-120 ms time window after the stimulus onset. By applying threshold-free cluster enhancement technique in combination with permutation statistics, we were able to detect the inverted IC effect-a relative suppression of the response to the Kanizsa figures compared with the control stimuli. The current finding is highly compatible with the explanation involving surround suppression evoked by iso-oriented collinear borders. The effect may be related to the principle of sparse coding, according to which V1 suppresses representations of inner parts of collinear assemblies as being informationally redundant. Such a mechanism is likely to be an important preliminary step preceding object contour detection.

Conjectures regarding the nonlinear geometry of visual neurons

From the earliest stages of sensory processing, neurons show inherent non-linearities: the response to a complex stimulus is not a sum of the responses to a set of constituent basis stimuli. These non-linearities come in a number of forms and have been explained in terms of a number of functional goals. The family of spatial non-linearities have included interactions that occur both within and outside of the classical receptive field. They include, saturation, cross orientation inhibition, contrast normalization, end-stopping and a variety of non-classical effects. In addition, neurons show a number of facilitatory and invariance related effects such as those exhibited by complex cells (integration across position). Here, we describe an approach that attempts to explain many of the non-linearities under a single geometric framework. In line with Zetzsche and colleagues (e.g., Zetzsche et al., 1999) we propose that many of the principal non-linearities can be described by a geometry where the neural response space has a simple curvature. In this paper, we focus on the geometry that produces both increased selectivity (curving outward) and increased tolerance (curving inward). We demonstrate that overcomplete sparse coding with both low-dimensional synthetic data and high-dimensional natural scene data can result in curvature that is responsible for a variety of different known non-classical effects including end-stopping and gain control. We believe that this approach provides a more fundamental explanation of these non-linearities and does not require that one postulate a variety of explanations (e.g., that gain must be controlled or the ends of lines must be detected). In its standard form, sparse coding does not however, produce invariance/tolerance represented by inward curvature. We speculate on some of the requirements needed to produce such curvature.

Boundary detection using double-opponency and spatial sparseness constraint

Brightness and color are two basic visual features integrated by the human visual system (HVS) to gain a better understanding of color natural scenes. Aiming to combine these two cues to maximize the reliability of boundary detection in natural scenes, we propose a new framework based on the color-opponent mechanisms of a certain type of color-sensitive double-opponent (DO) cells in the primary visual cortex (V1) of HVS. This type of DO cells has oriented receptive field with both chromatically and spatially opponent structure. The proposed framework is a feedforward hierarchical model, which has direct counterpart to the color-opponent mechanisms involved in from the retina to V1. In addition, we employ the spatial sparseness constraint (SSC) of neural responses to further suppress the unwanted edges of texture elements. Experimental results show that the DO cells we modeled can flexibly capture both the structured chromatic and achromatic boundaries of salient objects in complex scenes when the cone inputs to DO cells are unbalanced. Meanwhile, the SSC operator further improves the performance by suppressing redundant texture edges. With competitive contour detection accuracy, the proposed model has the additional advantage of quite simple implementation with low computational cost.

Modeling Inhibitory Interneurons in Efficient Sensory Coding Models

There is still much unknown regarding the computational role of inhibitory cells in the sensory cortex. While modeling studies could potentially shed light on the critical role played by inhibition in cortical computation, there is a gap between the simplicity of many models of sensory coding and the biological complexity of the inhibitory subpopulation. In particular, many models do not respect that inhibition must be implemented in a separate subpopulation, with those inhibitory interneurons having a diversity of tuning properties and characteristic E/I cell ratios. In this study we demonstrate a computational framework for implementing inhibition in dynamical systems models that better respects these biophysical observations about inhibitory interneurons. The main approach leverages recent work related to decomposing matrices into low-rank and sparse components via convex optimization, and explicitly exploits the fact that models and input statistics often have low-dimensional structure that can be exploited for efficient implementations. While this approach is applicable to a wide range of sensory coding models (including a family of models based on Bayesian inference in a linear generative model), for concreteness we demonstrate the approach on a network implementing sparse coding. We show that the resulting implementation stays faithful to the original coding goals while using inhibitory interneurons that are much more biophysically plausible.

Cortical function is a result of coordinated interactions between excitatory and inhibitory neural populations. In previous theoretical models of sensory systems, inhibitory neurons are often ignored or modeled too simplistically to contribute to understanding their role in cortical computation. In biophysical reality, inhibition is implemented with interneurons that have different characteristics from the population of excitatory cells. In this study, we propose a computational approach for including inhibition in theoretical models of neural coding in a way that respects several of these important characteristics, such as the relative number of inhibitory cells and the diversity of their response properties. The main idea is that the significant structure of the sensory world is reflected in very structured models of sensory coding, which can then be exploited in the implementation of the model using modern computational techniques. We demonstrate this approach on one specific model of sensory coding (called “sparse coding”) that has been successful at modeling other aspects of sensory cortex.

OPTIMAL SPARSE APPROXIMATION WITH INTEGRATE AND FIRE NEURONS

Sparse approximation is a hypothesized coding strategy where a population of sensory neurons (e.g. V1) encodes a stimulus using as few active neurons as possible. We present the Spiking LCA (locally competitive algorithm), a rate encoded Spiking Neural Network (SNN) of integrate and fire neurons that calculate sparse approximations. The Spiking LCA is designed to be equivalent to the nonspiking LCA, an analog dynamical system that converges on a ℓ1-norm sparse approximations exponentially. We show that the firing rate of the Spiking LCA converges on the same solution as the analog LCA, with an error inversely proportional to the sampling time. We simulate in NEURON a network of 128 neuron pairs that encode 8 × 8 pixel image patches, demonstrating that the network converges to nearly optimal encodings within 20 ms of biological time. We also show that when using more biophysically realistic parameters in the neurons, the gain function encourages additional ℓ0-norm sparsity in the encoding, relative both to ideal neurons and digital solvers.

Short-term memory capacity in networks via the restricted isometry property

Cortical networks are hypothesized to rely on transient network activity to support short-term memory (STM). In this letter, we study the capacity of randomly connected recurrent linear networks for performing STM when the input signals are approximately sparse in some basis. We leverage results from compressed sensing to provide rigorous nonasymptotic recovery guarantees, quantifying the impact of the input sparsity level, the input sparsity basis, and the network characteristics on the system capacity. Our analysis demonstrates that network memory capacities can scale superlinearly with the number of nodes and in some situations can achieve STM capacities that are much larger than the network size. We provide perfect recovery guarantees for finite sequences and recovery bounds for infinite sequences. The latter analysis predicts that network STM systems may have an optimal recovery length that balances errors due to omission and recall mistakes. Furthermore, we show that the conditions yielding optimal STM capacity can be embodied in several network topologies, including networks with sparse or dense connectivities.

Classification using sparse representations: a biologically plausible approach

Representing signals as linear combinations of basis vectors sparsely selected from an overcomplete dictionary has proven to be advantageous for many applications in pattern recognition, machine learning, signal processing, and computer vision. While this approach was originally inspired by insights into cortical information processing, biologically plausible approaches have been limited to exploring the functionality of early sensory processing in the brain, while more practical applications have employed non-biologically plausible sparse coding algorithms. Here, a biologically plausible algorithm is proposed that can be applied to practical problems. This algorithm is evaluated using standard benchmark tasks in the domain of pattern classification, and its performance is compared to a wide range of alternative algorithms that are widely used in signal and image processing. The results show that for the classification tasks performed here, the proposed method is competitive with the best of the alternative algorithms that have been evaluated. This demonstrates that classification using sparse representations can be performed in a neurally plausible manner, and hence, that this mechanism of classification might be exploited by the brain.