Do we know what the early visual system does?

Feature selection for inductive generalization

Judging similarities among objects, events, and experiences is one of the most basic cognitive abilities, allowing us to make predictions and generalizations. The main assumption in similarity judgment is that people selectively attend to salient features of stimuli and judge their similarities on the basis of the common and distinct features of the stimuli. However, it is unclear how people select features from stimuli and how they weigh features. Here, we present a computational method that helps address these questions. Our procedure combines image-processing techniques with a machine-learning algorithm and assesses feature weights that can account for both similarity and categorization judgment data. Our analysis suggests that a small number of local features are particularly important to explain our behavioral data.

In praise of "false" models and rich data

The authors argue that "true" models that aim at faithfully mimicking or reproducing every property of the sensorimotor system cannot be compact as they need many free parameters. Consequently, most scientists in motor control use what are called "false" models--models that derive from well-defined approximations. The authors conceptualize these models as a priori limited in scope and approximate. As such, they argue that a quantitative characterization of the deviations between the system and the model, more than the mere act of falsifying, allows scientists to make progress in understanding the sensorimotor system. Ultimately, this process should result in models that explain as much data variance as possible. The authors conclude by arguing that progress in that direction could strongly benefit from databases of experimental results and collections of models.

A single functional model accounts for the distinct properties of suppression in cortical area V1

Cross-orientation suppression and surround suppression have been extensively studied in primary visual cortex (V1). These two forms of suppression have some distinct properties which has led to the suggestion that they are generated by different underlying mechanisms. Furthermore, it has been suggested that mechanisms other than intracortical inhibition may be central to both forms of suppression. A simple computational model (PC/BC), in which intracortical inhibition is fundamental, is shown to simulate the distinct properties of cross-orientation and surround suppression. The same model has previously been shown to account for a large range of V1 response properties including orientation-tuning, spatial and temporal frequency tuning, facilitation and inhibition by flankers and textured surrounds as well as a range of other experimental results on cross-orientation suppression and surround suppression. The current results thus provide additional support for the PC/BC model of V1 and for the proposal that the diverse range of response properties observed in V1 neurons have a single computational explanation. Furthermore, these results demonstrate that current neurophysiological evidence is insufficient to discount intracortical inhibition as a central mechanism underlying both forms of suppression.

Beyond multiple pattern analyzers modeled as linear filters (as classical V1 simple cells): useful additions of the last 25 years

This review briefly discusses processes that have been suggested in the last 25 years as important to the intermediate stages of visual processing of patterns. Five categories of processes are presented: (1) Higher-order processes including FRF structures; (2) Divisive contrast nonlinearities including contrast normalization; (3) Subtractive contrast nonlinearities including contrast comparison; (4) Non-classical receptive fields (surround suppression, cross-orientation inhibition); (5) Contour integration.

A hypothesis of local intrinsic cortical signal processing

Nearly a century ago, Ramón y Cajal [1] speculated that cortical interneurones underlie specific functions that are fundamental to human thought. Here we develop a computational analysis of the function of local cortical loops and their synaptic connections. Specifically, we propose that the function of cortical interneurones is to reduce redundancy and to contribute to compute saliency of information represented in neurones by implementing divisive normalization and multiplicative filtering functions. This contextual filtering by cortical interneurones reduces the energy of locally homogeneous information flowing between different cortical areas, in a non-linear manner and along various event spaces, thereby ensuring a homeostatic level of informational selectivity. Dysregulations of the synaptic transmission in this ubiquitous basic building block of the functional architecture of the brain are correspondingly associated with disturbances of informational selectivity. Perturbations of synaptic transmission in local intrinsic connections of the cerebral cortex consequently lead to various kinds of cognitive and/or affective disorders, depending on the exact nature, the extension and the specific localization of the distortion.

The mechanics of embodiment: a dialog on embodiment and computational modeling

Embodied theories are increasingly challenging traditional views of cognition by arguing that conceptual representations that constitute our knowledge are grounded in sensory and motor experiences, and processed at this sensorimotor level, rather than being represented and processed abstractly in an amodal conceptual system. Given the established empirical foundation, and the relatively underspecified theories to date, many researchers are extremely interested in embodied cognition but are clamoring for more mechanistic implementations. What is needed at this stage is a push toward explicit computational models that implement sensorimotor grounding as intrinsic to cognitive processes. In this article, six authors from varying backgrounds and approaches address issues concerning the construction of embodied computational models, and illustrate what they view as the critical current and next steps toward mechanistic theories of embodiment. The first part has the form of a dialog between two fictional characters: Ernest, the "experimenter," and Mary, the "computational modeler." The dialog consists of an interactive sequence of questions, requests for clarification, challenges, and (tentative) answers, and touches the most important aspects of grounded theories that should inform computational modeling and, conversely, the impact that computational modeling could have on embodied theories. The second part of the article discusses the most important open challenges for embodied computational modeling.

Emergence of visual saliency from natural scenes via context-mediated probability distributions coding

Visual saliency is the perceptual quality that makes some items in visual scenes stand out from their immediate contexts. Visual saliency plays important roles in natural vision in that saliency can direct eye movements, deploy attention, and facilitate tasks like object detection and scene understanding. A central unsolved issue is: What features should be encoded in the early visual cortex for detecting salient features in natural scenes? To explore this important issue, we propose a hypothesis that visual saliency is based on efficient encoding of the probability distributions (PDs) of visual variables in specific contexts in natural scenes, referred to as context-mediated PDs in natural scenes. In this concept, computational units in the model of the early visual system do not act as feature detectors but rather as estimators of the context-mediated PDs of a full range of visual variables in natural scenes, which directly give rise to a measure of visual saliency of any input stimulus. To test this hypothesis, we developed a model of the context-mediated PDs in natural scenes using a modified algorithm for independent component analysis (ICA) and derived a measure of visual saliency based on these PDs estimated from a set of natural scenes. We demonstrated that visual saliency based on the context-mediated PDs in natural scenes effectively predicts human gaze in free-viewing of both static and dynamic natural scenes. This study suggests that the computation based on the context-mediated PDs of visual variables in natural scenes may underlie the neural mechanism in the early visual cortex for detecting salient features in natural scenes.

Mechanisms underlying the representation of angles embedded within contour stimuli in area V2 of macaque monkeys

We previously found that surprisingly many V2 neurons showed selective responses to particular angles embedded within continuous contours [M. Ito & H. Komatsu (2004)Journal of Neuroscience, 24, 3313-3324]. Here, we addressed whether the selectivity is dependent on the presence of individual constituent components or on the unique combination of these components. To reveal roles of constituent half-lines in response to whole angles, we conducted a quantitative model study after the framework of cascade models. Our linear-non-linear summation model implemented a few subunits selective to particular half-lines and was fitted to neuronal responses for each neuron. The study indicates that the best-fitting models well replicate the selectivity in the majority of V2 neurons and that the angle selectivity is dependent on a linear combination of responses to individual half-line components of the angles. The implication is that optimal angles are given by a combination of two preferred half-line components and the selectivity is sharpened by introducing suppression to non-preferred half-line components, rather than a specific facilitatory interaction between two preferred half-line components. The study indicates the participation of the gain control of responsiveness according to the number of half-line components. We also showed that the selectivity to acute angles depends on a combination of responses to one preferred component and weak responses to another component. Therefore, we concluded that the angle selectivity is dependent on selective responses to individual half-line components of the angles rather than a unique combination between them, whereas neurons could be selective to various angle widths at area V2.

Modulation of temporal precision in thalamic population responses to natural visual stimuli

Natural visual stimuli have highly structured spatial and temporal properties which influence the way visual information is encoded in the visual pathway. In response to natural scene stimuli, neurons in the lateral geniculate nucleus (LGN) are temporally precise - on a time scale of 10-25 ms - both within single cells and across cells within a population. This time scale, established by non stimulus-driven elements of neuronal firing, is significantly shorter than that of natural scenes, yet is critical for the neural representation of the spatial and temporal structure of the scene. Here, a generalized linear model (GLM) that combines stimulus-driven elements with spike-history dependence associated with intrinsic cellular dynamics is shown to predict the fine timing precision of LGN responses to natural scene stimuli, the corresponding correlation structure across nearby neurons in the population, and the continuous modulation of spike timing precision and latency across neurons. A single model captured the experimentally observed neural response, across different levels of contrasts and different classes of visual stimuli, through interactions between the stimulus correlation structure and the nonlinearity in spike generation and spike history dependence. Given the sensitivity of the thalamocortical synapse to closely timed spikes and the importance of fine timing precision for the faithful representation of natural scenes, the modulation of thalamic population timing over these time scales is likely important for cortical representations of the dynamic natural visual environment.

Recoding of sensory information across the retinothalamic synapse

The neural code that represents the world is transformed at each stage of a sensory pathway. These transformations enable downstream neurons to recode information they receive from earlier stages. Using the retinothalamic synapse as a model system, we developed a theoretical framework to identify stimulus features that are inherited, gained, or lost across stages. Specifically, we observed that thalamic spikes encode novel, emergent, temporal features not conveyed by single retinal spikes. Furthermore, we found that thalamic spikes are not only more informative than retinal ones, as expected, but also more independent. Next, we asked how thalamic spikes gain sensitivity to the emergent features. Explicitly, we found that the emergent features are encoded by retinal spike pairs and then recoded into independent thalamic spikes. Finally, we built a model of synaptic transmission that reproduced our observations. Thus, our results established a link between synaptic mechanisms and the recoding of sensory information.

Nonstimulated early visual areas carry information about surrounding context

Even within the early sensory areas, the majority of the input to any given cortical neuron comes from other cortical neurons. To extend our knowledge of the contextual information that is transmitted by such lateral and feedback connections, we investigated how visually nonstimulated regions in primary visual cortex (V1) and visual area V2 are influenced by the surrounding context. We used functional magnetic resonance imaging (fMRI) and pattern-classification methods to show that the cortical representation of a nonstimulated quarter-field carries information that can discriminate the surrounding visual context. We show further that the activity patterns in these regions are significantly related to those observed with feed-forward stimulation and that these effects are driven primarily by V1. These results thus demonstrate that visual context strongly influences early visual areas even in the absence of differential feed-forward thalamic stimulation.

Non-centered spike-triggered covariance analysis reveals neurotrophin-3 as a developmental regulator of receptive field properties of ON-OFF retinal ganglion cells

The functional separation of ON and OFF pathways, one of the fundamental features of the visual system, starts in the retina. During postnatal development, some retinal ganglion cells (RGCs) whose dendrites arborize in both ON and OFF sublaminae of the inner plexiform layer transform into RGCs with dendrites that monostratify in either the ON or OFF sublamina, acquiring final dendritic morphology in a subtype-dependent manner. Little is known about how the receptive field (RF) properties of ON, OFF, and ON-OFF RGCs mature during this time because of the lack of a reliable and efficient method to classify RGCs into these subtypes. To address this deficiency, we developed an innovative variant of Spike Triggered Covariance (STC) analysis, which we term Spike Triggered Covariance – Non-Centered (STC-NC) analysis. Using a multi-electrode array (MEA), we recorded the responses of a large population of mouse RGCs to a Gaussian white noise stimulus. As expected, the Spike-Triggered Average (STA) fails to identify responses driven by symmetric static nonlinearities such as those that underlie ON-OFF center RGC behavior. The STC-NC technique, in contrast, provides an efficient means to identify ON-OFF responses and quantify their RF center sizes accurately. Using this new tool, we find that RGCs gradually develop sensitivity to focal stimulation after eye opening, that the percentage of ON-OFF center cells decreases with age, and that RF centers of ON and ON-OFF cells become smaller. Importantly, we demonstrate for the first time that neurotrophin-3 (NT-3) regulates the development of physiological properties of ON-OFF center RGCs. Overexpression of NT-3 leads to the precocious maturation of RGC responsiveness and accelerates the developmental decrease of RF center size in ON-OFF cells. In summary, our study introduces STC-NC analysis which successfully identifies subtype RGCs and demonstrates how RF development relates to a neurotrophic driver in the retina.

The developmental separation of ON and OFF pathways is one of the fundamental features of the visual system. In the mouse retina, some bi-stratified ON-OFF RGCs are refined into mono-stratified ON or OFF RGCs during the first postnatal month. However, the process by which the RGCs' physiological receptive field properties mature remains incompletely characterized, mainly due to the lack of a reliable and efficient method to classify RGCs into different subtypes. Here we have developed an innovative analysis, Spike Triggered Covariance – Non-Centered (STC-NC), and demonstrated that this technique can accurately characterize the receptive field properties of ON, OFF and ON-OFF center cells. We show that, in wildtype mouse, RGCs gradually develop sensitivity to focal stimulation after eye opening, and the development of ON-OFF receptive field center properties correlates well with their dendritic laminar refinement. Furthermore, overexpression of NT-3 accelerates the developmental decrease of receptive field center size in ON-OFF cells. Our study is the first to establish the STC-NC analysis which can successfully identify ON-OFF subtype RGCs and to demonstrate how receptive field development relates to a neurotrophic driver in the retina.

Parallel processing of visual space by neighboring neurons in mouse visual cortex

Visual cortex shows smooth retinotopic organization on the macroscopic scale, but it is unknown how receptive fields are organized at the level of neighboring neurons. This information is crucial for discriminating among models of visual cortex. We used in vivo two-photon calcium imaging to independently map ON and OFF receptive field subregions of local populations of layer 2/3 neurons in mouse visual cortex. Receptive field subregions were often precisely shared among neighboring neurons. Furthermore, large subregions seem to be assembled from multiple smaller, non-overlapping subregions of other neurons in the same local population. These experiments provide, to our knowledge, the first characterization of the diversity of receptive fields in a dense local network of visual cortex and reveal elementary units of receptive field organization. Our results suggest that a limited pool of afferent receptive fields is available to a local population of neurons and reveal new organizational principles for the neural circuitry of the mouse visual cortex.

A generalized linear model of the impact of direct and indirect inputs to the lateral geniculate nucleus

Relay neurons in the lateral geniculate nucleus (LGN) receive direct visual input predominantly from a single retinal ganglion cell (RGC), in addition to indirect input from other sources including interneurons, thalamic reticular nucleus (TRN), and the visual cortex. To address the extent of influence of these indirect sources on the response properties of the LGN neurons, we fit a Generalized Linear Model (GLM) to the spike responses of cat LGN neurons driven by spatially homogeneous spots that were rapidly modulated by a pseudorandom luminance sequence. Several spot sizes were used to probe the spatial extent of the indirect visual effects. Our extracellular recordings captured both the LGN spikes and the incoming RGC input (S potentials), allowing us to divide the inputs to the GLM into two categories: the direct RGC input and the indirect input to which we have access through the luminance of the visual stimulus. For spots no larger than the receptive field center, the effect of the indirect input is negligible, while for larger spots its effect can, on average, account for 5% of the variance of the data and for as much as 25% in some cells. The polarity of the indirect visual influence is opposite to that of the linear receptive field of the neurons. We conclude that the indirect source of response modulation of the LGN relay neurons arises from inhibitory sources, compatible with thalamic interneurons or TRN.

Encoding and decoding in fMRI

Over the past decade fMRI researchers have developed increasingly sensitive techniques for analyzing the information represented in BOLD activity. The most popular of these techniques is linear classification, a simple technique for decoding information about experimental stimuli or tasks from patterns of activity across an array of voxels. A more recent development is the voxel-based encoding model, which describes the information about the stimulus or task that is represented in the activity of single voxels. Encoding and decoding are complementary operations: encoding uses stimuli to predict activity while decoding uses activity to predict information about the stimuli. However, in practice these two operations are often confused, and their respective strengths and weaknesses have not been made clear. Here we use the concept of a linearizing feature space to clarify the relationship between encoding and decoding. We show that encoding and decoding operations can both be used to investigate some of the most common questions about how information is represented in the brain. However, focusing on encoding models offers two important advantages over decoding. First, an encoding model can in principle provide a complete functional description of a region of interest, while a decoding model can provide only a partial description. Second, while it is straightforward to derive an optimal decoding model from an encoding model it is much more difficult to derive an encoding model from a decoding model. We propose a systematic modeling approach that begins by estimating an encoding model for every voxel in a scan and ends by using the estimated encoding models to perform decoding.

Complex cell receptive fields: evidence for a hierarchical mechanism

Simple cells in the primary visual cortex have segregated ON and OFF subregions in their receptive fields, while complex cells have overlapping ON and OFF subregions. These two cell types form the extremes at each end of a continuum of receptive field types. Hubel and Wiesel in 1962 suggested a hierarchical scheme of processing whereby spatially offset simple cells drive complex cells. Simple and complex cells are often classified by their responses to moving sine wave gratings: simple cells have oscillatory responses while complex cells produce unmodulated responses. Here, using moving gratings as stimuli, we show that a significant number of cells that display low levels of response modulation at high contrasts demonstrate high levels of response modulation at low contrasts. Most often a drifting low contrast grating generates a large phasic response at the fundamental frequency of the grating (F(1)) and a smaller but significant phasic response that is approximately 180 deg out-of-phase with the F(1) component. We present several models capable of capturing the effects of stimulus contrast on complex cell responses. The model that best reproduces our experimental results is a variation of the classical hierarchical model. In our model several spatially offset simple cells provide input to a complex cell, with each simple cell exhibiting a different contrast response function. At low contrasts only one of these simple cells is sufficiently excited to reveal its receptive field properties. As contrast is increased additional spatially offset simple cells with higher contrast thresholds add their responses to the overall spiking activity.

Biologically plausible saliency mechanisms improve feedforward object recognition

The biological plausibility of statistical inference and learning, tuned to the statistics of natural images, is investigated. It is shown that a rich family of statistical decision rules, confidence measures, and risk estimates, can be implemented with the computations attributed to the standard neurophysiological model of V1. In particular, different statistical quantities can be computed through simple re-arrangement of lateral divisive connections, non-linearities, and pooling. It is then shown that a number of proposals for the measurement of visual saliency can be implemented in a biologically plausible manner, through such re-arrangements. This enables the implementation of biologically plausible feedforward object recognition networks that include explicit saliency models. The potential of combined attention and recognition is illustrated by replacing the first layer of the HMAX architecture with a saliency network. Various saliency measures are compared, to investigate whether (1) saliency can substantially benefit visual recognition and (2) the benefits depend on the specific saliency mechanisms implemented. Experimental evaluation shows that saliency does indeed enhance recognition, but the gains are not independent of the saliency mechanisms. Best results are obtained with top-down mechanisms that equate saliency to classification confidence.

Applicability of White-Noise Techniques to Analyzing Motion Responses

Motion processing in visual neurons is often understood in terms of how they integrate light stimuli in space and time. These integrative properties, known as the spatiotemporal receptive fields (STRFs), are sometimes obtained using white-noise techniques where a continuous random contrast sequence is delivered to each spatial location within the cell's field of view. In contrast, motion stimuli such as moving bars are usually presented intermittently. Here we compare the STRF prediction of a neuron's response to a moving bar with the measured response in second-order interneurons (L-neurons) of dragonfly ocelli (simple eyes). These low-latency neurons transmit sudden changes in intensity and motion information to mediate flight and gaze stabilization reflexes. A white-noise analysis is made of the responses of L-neurons to random bar stimuli delivered either every frame (densely) or intermittently (sparsely) with a temporal sequence matched to the bar motion stimulus. Linear STRFs estimated using the sparse stimulus were significantly better at predicting the responses to moving bars than the STRFs estimated using a traditional dense white-noise stimulus, even when second-order nonlinear terms were added. Our results strongly suggest that visual adaptation significantly modifies the linear STRF properties of L-neurons in dragonfly ocelli during dense white-noise stimulation. We discuss the ability to predict the responses of visual neurons to arbitrary stimuli based on white-noise analysis. We also discuss the likely functional advantages that adaptive receptive field structures provide for stabilizing attitude during hover and forward flight in dragonflies.

Early suppressive mechanisms and the negative blood oxygenation level-dependent response in human visual cortex

Functional magnetic resonance imaging (fMRI) studies of early sensory cortex often measure stimulus-driven increases in the blood oxygenation level-dependent (BOLD) signal. However, these positive responses are frequently accompanied by reductions in the BOLD signal in adjacent regions of cortex. Although this negative BOLD response (NBR) is thought to result from neuronal suppression, the precise relationship between local activity, suppression, and perception remains unknown. By measuring BOLD signals in human primary visual cortex while varying the baseline contrast levels in the region affected by the NBR, we tested three physiologically plausible computational models of neuronal modulation that could explain this phenomenon: a subtractive model, a response gain model, and a contrast gain model. We also measured the ability of isoluminant contrast to generate an NBR. We show that the NBR can be modeled as a pathway-specific contrast gain modulation that is strongest outside the fovea. We found a similar spatial bias in a psychophysical study using identical stimuli, although these data indicated a response gain rather than a contrast gain mechanism. We reconcile these findings by proposing (1) that the NBR is associated with a long-range suppressive mechanism that hyperpolarizes a subset of magnocellularly driven neurons at the input to V1, (2) that this suppression is broadly tuned to match the spatial features of the mask region, and (3) that increasing the baseline contrast in the suppressed region drives all neurons in the input layer, reducing the relative contribution of the suppressing subpopulation in the fMRI signal.

Neural computation of visual imaging based on Kronecker product in the primary visual cortex

BACKGROUND

What kind of neural computation is actually performed by the primary visual cortex and how is this represented mathematically at the system level? It is an important problem in the visual information processing, but has not been well answered. In this paper, according to our understanding of retinal organization and parallel multi-channel topographical mapping between retina and primary visual cortex V1, we divide an image into orthogonal and orderly array of image primitives (or patches), in which each patch will evoke activities of simple cells in V1. From viewpoint of information processing, this activated process, essentially, involves optimal detection and optimal matching of receptive fields of simple cells with features contained in image patches. For the reconstruction of the visual image in the visual cortex V1 based on the principle of minimum mean squares error, it is natural to use the inner product expression in neural computation, which then is transformed into matrix form.

RESULTS

The inner product is carried out by using Kronecker product between patches and function architecture (or functional column) in localized and oriented neural computing. Compared with Fourier Transform, the mathematical description of Kronecker product is simple and intuitive, so is the algorithm more suitable for neural computation of visual cortex V1. Results of computer simulation based on two-dimensional Gabor pyramid wavelets show that the theoretical analysis and the proposed model are reasonable.

CONCLUSIONS

Our results are: 1. The neural computation of the retinal image in cortex V1 can be expressed to Kronecker product operation and its matrix form, this algorithm is implemented by the inner operation between retinal image primitives and primary visual cortex's column. It has simple, efficient and robust features, which is, therefore, such a neural algorithm, which can be completed by biological vision. 2. It is more suitable that the function of cortical column in cortex V1 is considered as the basic unit of visual image processing (such unit can implement basic multiplication of visual primitives, such as contour, line, and edge), rather than a set of tiled array filter. Fourier Transformation is replaced with Kronecker product, which greatly reduces the computational complexity. The neurobiological basis of this idea is that a visual image can be represented as a linear combination of orderly orthogonal primitive image containing some local feature. In the visual pathway, the image patches are topographically mapped onto cortex V1 through parallel multi-channels and then are processed independently by functional columns. Clearly, the above new perspective has some reference significance to exploring the neural mechanisms on the human visual information processing.

Synaptic and network mechanisms of sparse and reliable visual cortical activity during nonclassical receptive field stimulation

During natural vision, the entire visual field is stimulated by images rich in spatiotemporal structure. Although many visual system studies restrict stimuli to the classical receptive field (CRF), it is known that costimulation of the CRF and the surrounding nonclassical receptive field (nCRF) increases neuronal response sparseness. The cellular and network mechanisms underlying increased response sparseness remain largely unexplored. Here we show that combined CRF + nCRF stimulation increases the sparseness, reliability, and precision of spiking and membrane potential responses in classical regular spiking (RS(C)) pyramidal neurons of cat primary visual cortex. Conversely, fast-spiking interneurons exhibit increased activity and decreased selectivity during CRF + nCRF stimulation. The increased sparseness and reliability of RS(C) neuron spiking is associated with increased inhibitory barrages and narrower visually evoked synaptic potentials. Our experimental observations were replicated with a simple computational model, suggesting that network interactions among neuronal subtypes ultimately sharpen recurrent excitation, producing specific and reliable visual responses.

Learning to represent visual input

One of the central problems in computational neuroscience is to understand how the object-recognition pathway of the cortex learns a deep hierarchy of nonlinear feature detectors. Recent progress in machine learning shows that it is possible to learn deep hierarchies without requiring any labelled data. The feature detectors are learned one layer at a time and the goal of the learning procedure is to form a good generative model of images, not to predict the class of each image. The learning procedure only requires the pairwise correlations between the activations of neuron-like processing units in adjacent layers. The original version of the learning procedure is derived from a quadratic 'energy' function but it can be extended to allow third-order, multiplicative interactions in which neurons gate the pairwise interactions between other neurons. A technique for factoring the third-order interactions leads to a learning module that again has a simple learning rule based on pairwise correlations. This module looks remarkably like modules that have been proposed by both biologists trying to explain the responses of neurons and engineers trying to create systems that can recognize objects.

Representation of cross-frequency spatial phase relationships in human visual cortex

An image patch can be locally decomposed into sinusoidal waves of different orientations, spatial frequencies, amplitudes, and phases. The local phase information is essential for perception, because important visual features like edges emerge at locations of maximal local phase coherence. Detection of phase coherence requires integration of spatial frequency information across multiple spatial scales. Models of early visual processing suggest that the visual system should implement phase-sensitive pooling of spatial frequency information in the identification of broadband edges. We used functional magnetic resonance imaging (fMRI) adaptation to look for phase-sensitive neural responses in the human visual cortex. We found sensitivity to the phase difference between spatial frequency components in all studied visual areas, including the primary visual cortex (V1). Control experiments demonstrated that these results were not explained by differences in contrast or position. Next, we compared fMRI responses for broadband compound grating stimuli with congruent and random phase structures. All studied visual areas showed stronger responses for the stimuli with congruent phase structure. In addition, selectivity to phase congruency increased from V1 to higher-level visual areas along both the ventral and dorsal streams. We conclude that human V1 already shows phase-sensitive pooling of spatial frequencies, but only higher-level visual areas might be capable of pooling spatial frequency information across spatial scales typical for broadband natural images.

Overview of facts and issues about neural coding by spikes

In the present overview, our wish is to demystify some aspects of coding with spike-timing, through a simple review of well-understood technical facts regarding spike coding. Our goal is a better understanding of the extent to which computing and modeling with spiking neuron networks might be biologically plausible and computationally efficient. We intentionally restrict ourselves to a deterministic implementation of spiking neuron networks and we consider that the dynamics of a network is defined by a non-stochastic mapping. By staying in this rather simple framework, we are able to propose results, formula and concrete numerical values, on several topics: (i) general time constraints, (ii) links between continuous signals and spike trains, (iii) spiking neuron networks parameter adjustment. Beside an argued review of several facts and issues about neural coding by spikes, we propose new results, such as a numerical evaluation of the most critical temporal variables that schedule the progress of realistic spike trains. When implementing spiking neuron networks, for biological simulation or computational purpose, it is important to take into account the indisputable facts here unfolded. This precaution could prevent one from implementing mechanisms that would be meaningless relative to obvious time constraints, or from artificially introducing spikes when continuous calculations would be sufficient and more simple. It is also pointed out that implementing a large-scale spiking neuron network is finally a simple task.

Laminar and orientation-dependent characteristics of spatial nonlinearities: implications for the computational architecture of visual cortex

A full understanding of the computations performed in primary visual cortex is an important yet elusive goal. Receptive field models consisting of cascades of linear filters and static nonlinearities may be adequate to account for responses to simple stimuli such as gratings and random checkerboards, but their predictions of responses to complex stimuli such as natural scenes are only approximately correct. It is unclear whether these discrepancies are limited to quantitative inaccuracies that reflect well-recognized mechanisms such as response normalization, gain controls, and cross-orientation suppression or, alternatively, imply additional qualitative features of the underlying computations. To address this question, we examined responses of V1 and V2 neurons in the monkey and area 17 neurons in the cat to two-dimensional Hermite functions (TDHs). TDHs are intermediate in complexity between traditional analytic stimuli and natural scenes and have mathematical properties that facilitate their use to test candidate models. By exploiting these properties, along with the laminar organization of V1, we identify qualitative aspects of neural computations beyond those anticipated from the above-cited model framework. Specifically, we find that V1 neurons receive signals from orientation-selective mechanisms that are highly nonlinear: they are sensitive to phase correlations, not just spatial frequency content. That is, the behavior of V1 neurons departs from that of linear-nonlinear cascades with standard modulatory mechanisms in a qualitative manner: even relatively simple stimuli evoke responses that imply complex spatial nonlinearities. The presence of these findings in the input layers suggests that these nonlinearities act in a feedback fashion.

Bayesian reconstruction of natural images from human brain activity

Recent studies have used fMRI signals from early visual areas to reconstruct simple geometric patterns. Here, we demonstrate a new Bayesian decoder that uses fMRI signals from early and anterior visual areas to reconstruct complex natural images. Our decoder combines three elements: a structural encoding model that characterizes responses in early visual areas, a semantic encoding model that characterizes responses in anterior visual areas, and prior information about the structure and semantic content of natural images. By combining all these elements, the decoder produces reconstructions that accurately reflect both the spatial structure and semantic category of the objects contained in the observed natural image. Our results show that prior information has a substantial effect on the quality of natural image reconstructions. We also demonstrate that much of the variance in the responses of anterior visual areas to complex natural images is explained by the semantic category of the image alone.

Coding of stimulus sequences by population responses in visual cortex

Neuronal populations in sensory cortex represent the time-changing sensory input through a spatiotemporal code. What are the rules that govern this code? We measured membrane potentials and spikes from neuronal populations in cat visual cortex (V1), through voltage-sensitive dyes and electrode arrays. We first characterized the population response to a single orientation. As response amplitude grew, population tuning width remained constant for membrane potential responses and became progressively sharper for spike responses. We then asked how these single-orientation responses combine to code for successive orientations. We found that they combine through simple linear summation. Linearity, however, is violated after stimulus offset, when responses exhibit an unexplained persistence. Thanks to linearity, the interactions between responses to successive stimuli are minimal. We demonstrate that higher cortical areas may reconstruct the stimulus sequence from V1 population responses through a simple instantaneous decoder. In area V1, therefore, spatial and temporal coding operate largely independently.

Stimulus ensemble and cortical layer determine V1 spatial receptive fields

The concept of receptive field is a linear, feed-forward view of visual signal processing. Frequently used models of V1 neurons, like the dynamic Linear filter--static nonlinearity--Poisson [corrected] spike encoder model, predict that receptive fields measured with different stimulus ensembles should be similar. Here, we tested this concept by comparing spatiotemporal maps of V1 neurons derived from two very different, but commonly used, stimulus ensembles: sparse noise and Hartley subspace stimuli. We found maps from the two methods agreed for neurons in input layer 4C but were very different for neurons in superficial layers of V1. Many layer 2/3 cells have receptive fields with multiple elongated subregions when mapped with Hartley stimuli, but their spatial maps collapse to only a single, less-elongated subregion when mapped with sparse noise. Moreover, for upper layer V1 neurons, the preferred orientation for Hartley maps is much closer to the preferred orientation measured with drifting gratings than is the orientation preference of sparse-noise maps. These results challenge the concept of a stimulus-invariant receptive field and imply that intracortical interactions shape fundamental properties of layer 2/3 neurons.

Decision-theoretic saliency: computational principles, biological plausibility, and implications for neurophysiology and psychophysics

A decision-theoretic formulation of visual saliency, first proposed for top-down processing (object recognition) (Gao & Vasconcelos, 2005a), is extended to the problem of bottom-up saliency. Under this formulation, optimality is defined in the minimum probability of error sense, under a constraint of computational parsimony. The saliency of the visual features at a given location of the visual field is defined as the power of those features to discriminate between the stimulus at the location and a null hypothesis. For bottom-up saliency, this is the set of visual features that surround the location under consideration. Discrimination is defined in an information-theoretic sense and the optimal saliency detector derived for a class of stimuli that complies with known statistical properties of natural images. It is shown that under the assumption that saliency is driven by linear filtering, the optimal detector consists of what is usually referred to as the standard architecture of V1: a cascade of linear filtering, divisive normalization, rectification, and spatial pooling. The optimal detector is also shown to replicate the fundamental properties of the psychophysics of saliency: stimulus pop-out, saliency asymmetries for stimulus presence versus absence, disregard of feature conjunctions, and Weber's law. Finally, it is shown that the optimal saliency architecture can be applied to the solution of generic inference problems. In particular, for the class of stimuli studied, it performs the three fundamental operations of statistical inference: assessment of probabilities, implementation of Bayes decision rule, and feature selection.

Ideal observer analysis of signal quality in retinal circuits

The function of the retina is crucial, for it must encode visual signals so the brain can detect objects in the visual world. However, the biological mechanisms of the retina add noise to the visual signal and therefore reduce its quality and capacity to inform about the world. Because an organism's survival depends on its ability to unambiguously detect visual stimuli in the presence of noise, its retinal circuits must have evolved to maximize signal quality, suggesting that each retinal circuit has a specific functional role. Here we explain how an ideal observer can measure signal quality to determine the functional roles of retinal circuits. In a visual discrimination task the ideal observer can measure from a neural response the increment threshold, the number of distinguishable response levels, and the neural code, which are fundamental measures of signal quality relevant to behavior. It can compare the signal quality in stimulus and response to determine the optimal stimulus, and can measure the specific loss of signal quality by a neuron's receptive field for non-optimal stimuli. Taking into account noise correlations, the ideal observer can track the signal-to-noise ratio available from one stage to the next, allowing one to determine each stage's role in preserving signal quality. A comparison between the ideal performance of the photon flux absorbed from the stimulus and actual performance of a retinal ganglion cell shows that in daylight a ganglion cell and its presynaptic circuit loses a factor of approximately 10-fold in contrast sensitivity, suggesting specific signal-processing roles for synaptic connections and other neural circuit elements. The ideal observer is a powerful tool for characterizing signal processing in single neurons and arrays along a neural pathway.

Camouflage and visual perception

How does an animal conceal itself from visual detection by other animals? This review paper seeks to identify general principles that may apply in this broad area. It considers mechanisms of visual encoding, of grouping and object encoding, and of search. In most cases, the evidence base comes from studies of humans or species whose vision approximates to that of humans. The effort is hampered by a relatively sparse literature on visual function in natural environments and with complex foraging tasks. However, some general constraints emerge as being potentially powerful principles in understanding concealment--a 'constraint' here means a set of simplifying assumptions. Strategies that disrupt the unambiguous encoding of discontinuities of intensity (edges), and of other key visual attributes, such as motion, are key here. Similar strategies may also defeat grouping and object-encoding mechanisms. Finally, the paper considers how we may understand the processes of search for complex targets in complex scenes. The aim is to provide a number of pointers towards issues, which may be of assistance in understanding camouflage and concealment, particularly with reference to how visual systems can detect the shape of complex, concealed objects.

Modeling convergent ON and OFF pathways in the early visual system

For understanding the computation and function of single neurons in sensory systems, one needs to investigate how sensory stimuli are related to a neuron's response and which biological mechanisms underlie this relationship. Mathematical models of the stimulus-response relationship have proved very useful in approaching these issues in a systematic, quantitative way. A starting point for many such analyses has been provided by phenomenological "linear-nonlinear" (LN) models, which comprise a linear filter followed by a static nonlinear transformation. The linear filter is often associated with the neuron's receptive field. However, the structure of the receptive field is generally a result of inputs from many presynaptic neurons, which may form parallel signal processing pathways. In the retina, for example, certain ganglion cells receive excitatory inputs from ON-type as well as OFF-type bipolar cells. Recent experiments have shown that the convergence of these pathways leads to intriguing response characteristics that cannot be captured by a single linear filter. One approach to adjust the LN model to the biological circuit structure is to use multiple parallel filters that capture ON and OFF bipolar inputs. Here, we review these new developments in modeling neuronal responses in the early visual system and provide details about one particular technique for obtaining the required sets of parallel filters from experimental data.

Nonlinear modeling of causal interrelationships in neuronal ensembles

The increasing availability of multiunit recordings gives new urgency to the need for effective analysis of "multidimensional" time-series data that are derived from the recorded activity of neuronal ensembles in the form of multiple sequences of action potentials--treated mathematically as point-processes and computationally as spike-trains. Whether in conditions of spontaneous activity or under conditions of external stimulation, the objective is the identification and quantification of possible causal links among the neurons generating the observed binary signals. A multiple-input/multiple-output (MIMO) modeling methodology is presented that can be used to quantify the neuronal dynamics of causal interrelationships in neuronal ensembles using spike-train data recorded from individual neurons. These causal interrelationships are modeled as transformations of spike-trains recorded from a set of neurons designated as the "inputs" into spike-trains recorded from another set of neurons designated as the "outputs." The MIMO model is composed of a set of multiinput/single-output (MISO) modules, one for each output. Each module is the cascade of a MISO Volterra model and a threshold operator generating the output spikes. The Laguerre expansion approach is used to estimate the Volterra kernels of each MISO module from the respective input-output data using the least-squares method. The predictive performance of the model is evaluated with the use of the receiver operating characteristic (ROC) curve, from which the optimum threshold is also selected. The Mann-Whitney statistic is used to select the significant inputs for each output by examining the statistical significance of improvements in the predictive accuracy of the model when the respective inputs is included. Illustrative examples are presented for a simulated system and for an actual application using multiunit data recordings from the hippocampus of a behaving rat.

Distinct expressions of contrast gain control in parallel synaptic pathways converging on a retinal ganglion cell

Visual neurons adapt to increases in stimulus contrast by reducing their response sensitivity and decreasing their integration time, a collective process known as 'contrast gain control.' In retinal ganglion cells, gain control arises at two stages: an intrinsic mechanism related to spike generation, and a synaptic mechanism in retinal pathways. Here, we tested whether gain control is expressed similarly by three synaptic pathways that converge on an OFF alpha/Y-type ganglion cell: excitatory inputs driven by OFF cone bipolar cells; inhibitory inputs driven by ON cone bipolar cells; and inhibitory inputs driven by rod bipolar cells. We made whole-cell recordings of membrane current in guinea pig ganglion cells in vitro. At high contrast, OFF bipolar cell-mediated excitatory input reduced gain and shortened integration time. Inhibitory input was measured by clamping voltage near 0 mV or by recording in the presence of ionotropic glutamate receptor (iGluR) antagonists to isolate the following circuit: cone --> ON cone bipolar cell --> AII amacrine cell --> OFF ganglion cell. At high contrast, this input reduced gain with no effect on integration time. Mean luminance was reduced 1000-fold to recruit the rod bipolar pathway: rod --> rod bipolar cell --> AII cell --> OFF ganglion cell. The spiking response, measured with loose-patch recording, adapted despite essentially no gain control in synaptic currents. Thus, cone bipolar-driven pathways adapt differently, with kinetic effects confined to the excitatory OFF pathway. The ON bipolar-mediated inhibition reduced gain at high contrast by a mechanism that did not require an iGluR. Under rod bipolar-driven conditions, ganglion cell firing showed gain control that was explained primarily by an intrinsic property.

Highly selective receptive fields in mouse visual cortex

Genetic methods available in mice are likely to be powerful tools in dissecting cortical circuits. However, the visual cortex, in which sensory coding has been most thoroughly studied in other species, has essentially been neglected in mice perhaps because of their poor spatial acuity and the lack of columnar organization such as orientation maps. We have now applied quantitative methods to characterize visual receptive fields in mouse primary visual cortex V1 by making extracellular recordings with silicon electrode arrays in anesthetized mice. We used current source density analysis to determine laminar location and spike waveforms to discriminate putative excitatory and inhibitory units. We find that, although the spatial scale of mouse receptive fields is up to one or two orders of magnitude larger, neurons show selectivity for stimulus parameters such as orientation and spatial frequency that is near to that found in other species. Furthermore, typical response properties such as linear versus nonlinear spatial summation (i.e., simple and complex cells) and contrast-invariant tuning are also present in mouse V1 and correlate with laminar position and cell type. Interestingly, we find that putative inhibitory neurons generally have less selective, and nonlinear, responses. This quantitative description of receptive field properties should facilitate the use of mouse visual cortex as a system to address longstanding questions of visual neuroscience and cortical processing.

Virtual Retina: a biological retina model and simulator, with contrast gain control

We propose a new retina simulation software, called Virtual Retina, which transforms a video into spike trains. Our goal is twofold: Allow large scale simulations (up to 100,000 neurons) in reasonable processing times and keep a strong biological plausibility, taking into account implementation constraints. The underlying model includes a linear model of filtering in the Outer Plexiform Layer, a shunting feedback at the level of bipolar cells accounting for rapid contrast gain control, and a spike generation process modeling ganglion cells. We prove the pertinence of our software by reproducing several experimental measurements from single ganglion cells such as cat X and Y cells. This software will be an evolutionary tool for neuroscientists that need realistic large-scale input spike trains in subsequent treatments, and for educational purposes.

A two-stage neural spiking model of visual contrast detection in perimetry

Perimetry is a commonly used clinical test for visual function, limited by high variability. The sources of this variability need to be better understood. In this paper, we investigate whether noise intrinsic to neural firing could explain the variability in normal subjects. We present the most physiologically accurate model to date for stimulus detection in perimetry combining knowledge of the physiology of components of the visual system with signal detection theory, and show that it requires that detection be mediated by multiple cortical cells in order to give predictions consistent with psychometric functions measured in human observers.

Non-linear population firing rates and voltage sensitive dye signals in visual areas 17 and 18 to short duration stimuli

Visual stimuli of short duration seem to persist longer after the stimulus offset than stimuli of longer duration. This visual persistence must have a physiological explanation. In ferrets exposed to stimuli of different durations we measured the relative changes in the membrane potentials with a voltage sensitive dye and the action potentials of populations of neurons in the upper layers of areas 17 and 18. For durations less than 100 ms, the timing and amplitude of the firing and membrane potentials showed several non-linear effects. The ON response became truncated, the OFF response progressively reduced, and the timing of the OFF responses progressively delayed the shorter the stimulus duration. The offset of the stimulus elicited a sudden and strong negativity in the time derivative of the dye signal. All these non-linearities could be explained by the stimulus offset inducing a sudden inhibition in layers II-III as indicated by the strongly negative time derivative of the dye signal. Despite the non-linear behavior of the layer II-III neurons the sum of the action potentials, integrated from the peak of the ON response to the peak of the OFF response, was almost linearly related to the stimulus duration.

Functional circuitry of visual adaptation in the retina

The visual system continually adjusts its sensitivity, or 'adapts', to the conditions of the immediate environment. Adaptation increases responses when input signals are weak, to improve the signal-to-noise ratio, and decreases responses when input signals are strong, to prevent response saturation. Retinal ganglion cells adapt primarily to two properties of light input: the mean intensity and the variance of intensity over time (contrast). This review focuses on cellular mechanisms for contrast adaptation in mammalian retina. High contrast over the ganglion cell's receptive field centre reduces the gain of spiking responses. The mechanism for gain control arises partly in presynaptic bipolar cell inputs and partly in the process of spike generation. Following strong contrast stimulation, ganglion cells exhibit a prolonged after-hyperpolarization, driven primarily by suppression of glutamate release from presynaptic bipolar cells. Ganglion cells also adapt to high contrast over their peripheral receptive field. Long-range adaptive signals are carried by amacrine cells that inhibit the ganglion cell directly, causing hyperpolarization, and inhibit presynaptic bipolar terminals, reducing gain of their synaptic output. Thus, contrast adaptation in ganglion cells involves multiple synaptic and intrinsic mechanisms for gain control and hyperpolarization. Several forms of adaptation in ganglion cells originate in presynaptic bipolar cells.

Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli

Functional models of the early visual system should predict responses not only to simple artificial stimuli but also to sequences of complex natural scenes. An ideal testbed for such models is the lateral geniculate nucleus (LGN). Mechanisms shaping LGN responses include the linear receptive field and two fast adaptation processes, sensitive to luminance and contrast. We propose a compact functional model for these mechanisms that operates on sequences of arbitrary images. With the same parameters that fit the firing rate responses to simple stimuli, it predicts the bulk of the firing rate responses to complex stimuli, including natural scenes. Further improvements could result by adding a spiking mechanism, possibly one capable of bursts, but not by adding mechanisms of slow adaptation. We conclude that up to the LGN the responses to natural scenes can be largely explained through insights gained with simple artificial stimuli.

Saccadic facilitation in natural backgrounds

In visual systems with a fovea, only a small portion of the visual field can be analyzed with high accuracy. Saccadic eye movements shift that center of gaze around several times a second. Saccades have been characterized in great detail and depend critically on a number of visual properties of the stimuli. However, typical experiments have used bright spots on dark backgrounds, while our natural environment has a highly characteristic rich spatial structure. Here we show that the saccadic system, unlike the perceptual system, is able to compensate for the masking caused by structured backgrounds. Consequently, saccadic latencies in the context of natural backgrounds are much faster than unstructured backgrounds at equal levels of visibility. The results suggest that whenever a structured background acts to mask the visibility of the saccade target, it simultaneously preactivates saccadic circuitry and thus ensures a fast reaction to potentially critical stimuli that are difficult to detect in our environment.

Fixational eye movements, natural image statistics, and fine spatial vision

Perception and motor control are often regarded as two separate branches of neuroscience. Like most species, however, humans are not passively exposed to the incoming flow of sensory data, but actively seek useful information. By shaping input signals in ways that simplify perceptual tasks, behavior might play an important role in establishing efficient sensory representations in the brain. Under natural viewing conditions, the main source of motion of the stimulus on the retina is not the scene but our own behavior. The retinal image is never still, even during visual fixation, when small eye movements combine with movements of the head and body to continually perturb the location of gaze. This article examines the impact of the fixational motion of the retinal image on the statistics of visual input and the neural encoding of visual information. Building upon recent theoretical and experimental results, it is argued that an unstable fixation constitutes an efficient strategy for acquiring information from natural scenes. According to this theory, the fluctuations of luminance caused by the incessant motion of the eye equalize the power present at different spatial frequencies in the spatiotemporal stimulus on the retina. This phenomenon yields compact neural representations, emphasizes fine spatial detail, and might enable a temporal multiplexing of visual information from the retina to the cortex. This theory posits motor contributions to early visual representations and suggests that perception and behavior are more intimately tied than commonly thought.

Modeling spatial integration in the ocular following response using a probabilistic framework

The machinery behind the visual perception of motion and the subsequent sensori-motor transformation, such as in ocular following response (OFR), is confronted to uncertainties which are efficiently resolved in the primate's visual system. We may understand this response as an ideal observer in a probabilistic framework by using Bayesian theory [Weiss, Y., Simoncelli, E.P., Adelson, E.H., 2002. Motion illusions as optimal percepts. Nature Neuroscience, 5(6), 598-604, doi:10.1038/nn858] which we previously proved to be successfully adapted to model the OFR for different levels of noise with full field gratings. More recent experiments of OFR have used disk gratings and bipartite stimuli which are optimized to study the dynamics of center-surround integration. We quantified two main characteristics of the spatial integration of motion: (i) a finite optimal stimulus size for driving OFR, surrounded by an antagonistic modulation and (ii) a direction selective suppressive effect of the surround on the contrast gain control of the central stimuli [Barthélemy, F.V., Vanzetta, I., Masson, G.S., 2006. Behavioral receptive field for ocular following in humans: dynamics of spatial summation and center-surround interactions. Journal of Neurophysiology, (95), 3712-3726, doi:10.1152/jn.00112.2006]. Herein, we extended the ideal observer model to simulate the spatial integration of the different local motion cues within a probabilistic representation. We present analytical results which show that the hypothesis of independence of local measures can describe the spatial integration of the motion signal. Within this framework, we successfully accounted for the contrast gain control mechanisms observed in the behavioral data for center-surround stimuli. However, another inhibitory mechanism had to be added to account for suppressive effects of the surround.

Receptive field for dorsal cochlear nucleus neurons at multiple sound levels

Neurons in the dorsal cochlear nucleus (DCN) exhibit nonlinearities in spectral processing, which make it difficult to predict the neurons' responses to stimuli. Here, we consider two possible sources of nonlinearity: nonmonotonic responses as sound level increases due to inhibition and interactions between frequency components. A spectral weighting function model of rate responses is used; the model approximates the neuron's rate response as a weighted sum of the frequency components of the stimulus plus a second-order sum that captures interactions between frequencies. Such models approximate DCN neurons well at low spectral contrast, i.e., when the SD (contrast) of the stimulus spectrum is limited to 3 dB. This model is compared with a first-order sum with weights that are explicit functions of sound level, so that the low-contrast model is extended to spectral contrasts of 12 dB, the range of natural stimuli. The sound-level-dependent weights improve prediction performance at large spectral contrast. However, the interactions between frequencies, represented as second-order terms, are more important at low spectral contrast. The level-dependent model is shown to predict previously described patterns of responses to spectral edges, showing that small changes in the inhibitory components of the receptive field can produce large changes in the responses of the neuron to features of natural stimuli. These results provide an effective way of characterizing nonlinear auditory neurons incorporating stimulus-dependent sensitivity changes. Such models could be used for neurons in other sensory systems that show similar effects.