Delayed suppression shapes disparity selective responses in monkey V1

Model-based characterization of the selectivity of neurons in primary visual cortex

Statistical models are increasingly being used to understand the complexity of stimulus selectivity in primary visual cortex (V1) in the context of complex time-varying stimuli, replacing averaging responses to simple parametric stimuli. Although such models often can more accurately reflect the computations performed by V1 neurons in more natural visual environments, they do not by themselves provide insight into V1 neural selectivity to basic stimulus features such as receptive field size, spatial frequency tuning, and phase invariance. Here, we present a battery of analyses that can be directly applied to encoding models to link complex encoding models to more interpretable aspects of stimulus selectivity. We apply this battery to nonlinear models of V1 neurons recorded in awake macaque during random bar stimuli. In linking model properties to more classical measurements, we demonstrate several novel aspects of V1 selectivity not available to simpler experimental measurements. For example, this approach reveals that individual spatiotemporal elements of the V1 models often have a smaller spatial scale than the neuron as a whole, resulting in nontrivial tuning to spatial frequencies. In addition, we propose measures of nonlinear integration that suggest that classical classifications of V1 neurons into simple versus complex cells will be spatial-frequency dependent. In total, rather than obfuscate classical characterizations of V1 neurons, model-based characterizations offer a means to more fully understand their selectivity, and link their classical tuning properties to their roles in more complex, natural, visual processing.NEW & NOTEWORTHY Visual neurons are increasingly being studied with more complex, natural visual stimuli, and increasingly complex models are necessary to characterize their response properties. Here, we describe a battery of analyses that relate these more complex models to classical characterizations. Using such model-based characterizations of V1 neurons furthermore yields several new insights into V1 processing not possible to capture in more classical means to measure their visual selectivity.

Ultra-High-Field Neuroimaging Reveals Fine-Scale Processing for 3D Perception

Binocular disparity provides critical information about three-dimensional (3D) structures to support perception and action. In the past decade significant progress has been made in uncovering human brain areas engaged in the processing of binocular disparity signals. Yet, the fine-scale brain processing underlying 3D perception remains unknown. Here, we use ultra-high-field (7T) functional imaging at submillimeter resolution to examine fine-scale BOLD fMRI signals involved in 3D perception. In particular, we sought to interrogate the local circuitry involved in disparity processing by sampling fMRI responses at different positions relative to the cortical surface (i.e., across cortical depths corresponding to layers). We tested for representations related to 3D perception by presenting participants (male and female, N = 8) with stimuli that enable stable stereoscopic perception [i.e., correlated random dot stereograms (RDS)] versus those that do not (i.e., anticorrelated RDS). Using multivoxel pattern analysis (MVPA), we demonstrate cortical depth-specific representations in areas V3A and V7 as indicated by stronger pattern responses for correlated than for anticorrelated stimuli in upper rather than deeper layers. Examining informational connectivity, we find higher feedforward layer-to-layer connectivity for correlated than anticorrelated stimuli between V3A and V7. Further, we observe disparity-specific feedback from V3A to V1 and from V7 to V3A. Our findings provide evidence for the role of V3A as a key nexus for disparity processing, which is implicated in feedforward and feedback signals related to the perceptual estimation of 3D structures.SIGNIFICANCE STATEMENT Binocular vision plays a significant role in supporting our interactions with the surrounding environment. The fine-scale neural mechanisms that underlie the brain's skill in extracting 3D structures from binocular signals are poorly understood. Here, we capitalize on recent advances in ultra-high-field functional imaging to interrogate human brain circuits involved in 3D perception at submillimeter resolution. We provide evidence for the role of area V3A as a key nexus for disparity processing, which is implicated in feedforward and feedback signals related to the perceptual estimation of 3D structures from binocular signals. These fine-scale measurements help bridge the gap between animal neurophysiology and human fMRI studies investigating cross-scale circuits, from micro circuits to global brain networks for 3D perception.

Exploitation of image statistics with sparse coding in the case of stereo vision

The sparse coding algorithm has served as a model for early processing in mammalian vision. It has been assumed that the brain uses sparse coding to exploit statistical properties of the sensory stream. We hypothesize that sparse coding discovers patterns from the data set, which can be used to estimate a set of stimulus parameters by simple readout. In this study, we chose a model of stereo vision to test our hypothesis. We used the Locally Competitive Algorithm (LCA), followed by a naïve Bayes classifier, to infer stereo disparity. From the results we report three observations. First, disparity inference was successful with this naturalistic processing pipeline. Second, an expanded, highly redundant representation is required to robustly identify the input patterns. Third, the inference error can be predicted from the number of active coefficients in the LCA representation. We conclude that sparse coding can generate a suitable general representation for subsequent inference tasks.

Distinct spatiotemporal mechanisms underlie extra-classical receptive field modulation in macaque V1 microcircuits

Complex scene perception depends upon the interaction between signals from the classical receptive field (CRF) and the extra-classical receptive field (eCRF) in primary visual cortex (V1) neurons. Although much is known about V1 eCRF properties, we do not yet know how the underlying mechanisms map onto the cortical microcircuit. We probed the spatio-temporal dynamics of eCRF modulation using a reverse correlation paradigm, and found three principal eCRF mechanisms: tuned-facilitation, untuned-suppression, and tuned-suppression. Each mechanism had a distinct timing and spatial profile. Laminar analysis showed that the timing, orientation-tuning, and strength of eCRF mechanisms had distinct signatures within magnocellular and parvocellular processing streams in the V1 microcircuit. The existence of multiple eCRF mechanisms provides new insights into how V1 responds to spatial context. Modeling revealed that the differences in timing and scale of these mechanisms predicted distinct patterns of net modulation, reconciling many previous disparate physiological and psychophysical findings.

Temporal dynamics of binocular integration in primary visual cortex

Whenever we open our eyes, our brain quickly integrates the two eyes' perspectives into a combined view. This process of binocular integration happens so rapidly that even incompatible stimuli are briefly fused before one eye's view is suppressed in favor of the other (binocular rivalry). The neuronal basis for this brief period of fusion during incompatible binocular stimulation is unclear. Neuroanatomically, the eyes provide two largely separate streams of information that are integrated into a binocular response by the primary visual cortex (V1). However, the temporal dynamics underlying the formation of this binocular response are largely unknown. To address this question, we examined the temporal profile of binocular responses in V1 of fixating monkeys. We found that V1 processes binocular stimuli in a dynamic sequence that comprises at least two distinct temporal phases. An initial transient phase is characterized by enhanced spiking responses for both compatible and incompatible binocular stimuli compared to monocular stimulation. This transient is followed by a sustained response that differed markedly between congruent and incongruent binocular stimulation. Specifically, incompatible binocular stimulation resulted in overall response reduction relative to monocular stimulation (binocular suppression). In contrast, responses to compatible stimuli were either suppressed or enhanced (binocular facilitation) depending on the neurons' ocularity (selectivity for one eye over the other) and laminar location. These results suggest that binocular integration in V1 occurs in at least two sequential steps that comprise initial additive combination of the two eyes' signals followed by widespread differentiation between binocular concordance and discordance.

Visual Perception: Neural Networks for Stereopsis

How does our brain use differences between the images in our two eyes, binocular disparities, to generate depth perception? New work shows that a type of neural network trained on natural binocular images can learn parameters that match key properties of visual cortex. Most information is conveyed by cells which sense differences between the two eyes' images.

Disparity processing in primary visual cortex

The first step in binocular stereopsis is to match features on the left retina with the correct features on the right retina, discarding 'false' matches. The physiological processing of these signals starts in the primary visual cortex, where the binocular energy model has been a powerful framework for understanding the underlying computation. For this reason, it is often used when thinking about how binocular matching might be performed beyond striate cortex. But this step depends critically on the accuracy of the model, and real V1 neurons show several properties that suggest they may be less sensitive to false matches than the energy model predicts. Several recent studies provide empirical support for an extended version of the energy model, in which the same principles are used, but the responses of single neurons are described as the sum of several subunits, each of which follows the principles of the energy model. These studies have significantly improved our understanding of the role played by striate cortex in the stereo correspondence problem.This article is part of the themed issue 'Vision in our three-dimensional world'.

Neurons in Striate Cortex Signal Disparity in Half-Matched Random-Dot Stereograms

Human stereopsis can operate in dense "cyclopean" images containing no monocular objects. This is believed to depend on the computation of binocular correlation by neurons in primary visual cortex (V1). The observation that humans perceive depth in half-matched random-dot stereograms, although these stimuli have no net correlation, has led to the proposition that human depth perception in these stimuli depends on a distinct "matching" computation possibly performed in extrastriate cortex. However, recording from disparity-selective neurons in V1 of fixating monkeys, we found that they are in fact able to signal disparity in half-matched stimuli. We present a simple model that explains these results. This reinstates the view that disparity-selective neurons in V1 provide the initial substrate for perception in dense cyclopean stimuli, and strongly suggests that separate correlation and matching computations are not necessary to explain existing data on mixed correlation stereograms.

SIGNIFICANCE STATEMENT

The initial step in stereoscopic 3D vision is generally thought to be a correlation-based computation that takes place in striate cortex. Recent research has argued that there must be an additional matching computation involved in extracting stereoscopic depth in random-dot stereograms. This is based on the observation that humans can perceive depth in stimuli with a mean binocular correlation of zero (where a correlation-based mechanism should not signal depth). We show that correlation-based cells in striate cortex do in fact signal depth here because they convert fluctuations in the correlation level into a mean change in the firing rate. Our results reinstate the view that these cells provide a sufficient substrate for the perception of stereoscopic depth.

Effects of generalized pooling on binocular disparity selectivity of neurons in the early visual cortex

The key problem of stereoscopic vision is traditionally defined as accurately finding the positional shifts of corresponding object features between left and right images. Here, we demonstrate that the problem must be considered in a four-dimensional parameter space; with respect not only to shifts in space (X, Y), but also spatial frequency (SF) and orientation (OR). The proposed model sums outputs of binocular energy units linearly over the multi-dimensional V1 parameter space (X, Y, SF, OR). Theoretical analyses and physiological experiments show that many binocular neurons achieve sharp binocular tuning properties by pooling the output of multiple neurons with relatively broad tuning. Pooling in the space domain sharpens disparity-selective responses in the SF domain so that the responses to combinations of unmatched left-right SFs are attenuated. Conversely, pooling in the SF domain sharpens disparity selectivity in the space domain, reducing the possibility of false matches. Analogous effects are observed for the OR domain in that the spatial pooling sharpens the binocular tuning in the OR domain. Such neurons become selective to relative OR disparity. Therefore, pooling allows the visual system to refine binocular information into a form more desirable for stereopsis.This article is part of the themed issue 'Vision in our three-dimensional world'.

"What Not" Detectors Help the Brain See in Depth

Binocular stereopsis is one of the primary cues for three-dimensional (3D) vision in species ranging from insects to primates. Understanding how the brain extracts depth from two different retinal images represents a tractable challenge in sensory neuroscience that has so far evaded full explanation. Central to current thinking is the idea that the brain needs to identify matching features in the two retinal images (i.e., solving the “stereoscopic correspondence problem”) so that the depth of objects in the world can be triangulated. Although intuitive, this approach fails to account for key physiological and perceptual observations. We show that formulating the problem to identify “correct matches” is suboptimal and propose an alternative, based on optimal information encoding, that mixes disparity detection with “proscription”: exploiting dissimilar features to provide evidence against unlikely interpretations. We demonstrate the role of these “what not” responses in a neural network optimized to extract depth in natural images. The network combines information for and against the likely depth structure of the viewed scene, naturally reproducing key characteristics of both neural responses and perceptual interpretations. We capture the encoding and readout computations of the network in simple analytical form and derive a binocular likelihood model that provides a unified account of long-standing puzzles in 3D vision at the physiological and perceptual levels. We suggest that marrying detection with proscription provides an effective coding strategy for sensory estimation that may be useful for diverse feature domains (e.g., motion) and multisensory integration.

•

The brain uses “what not” detectors to facilitate 3D vision

•

Binocular mismatches are used to drive suppression of incompatible depths

•

Proscription accounts for depth perception without binocular correspondence

•

A simple analytical model captures perceptual and neural responses

Cross-matching: a modified cross-correlation underlying threshold energy model and match-based depth perception

Three-dimensional visual perception requires correct matching of images projected to the left and right eyes. The matching process is faced with an ambiguity: part of one eye's image can be matched to multiple parts of the other eye's image. This stereo correspondence problem is complicated for random-dot stereograms (RDSs), because dots with an identical appearance produce numerous potential matches. Despite such complexity, human subjects can perceive a coherent depth structure. A coherent solution to the correspondence problem does not exist for anticorrelated RDSs (aRDSs), in which luminance contrast is reversed in one eye. Neurons in the visual cortex reduce disparity selectivity for aRDSs progressively along the visual processing hierarchy. A disparity-energy model followed by threshold nonlinearity (threshold energy model) can account for this reduction, providing a possible mechanism for the neural matching process. However, the essential computation underlying the threshold energy model is not clear. Here, we propose that a nonlinear modification of cross-correlation, which we term “cross-matching,” represents the essence of the threshold energy model. We placed half-wave rectification within the cross-correlation of the left-eye and right-eye images. The disparity tuning derived from cross-matching was attenuated for aRDSs. We simulated a psychometric curve as a function of graded anticorrelation (graded mixture of aRDS and normal RDS); this simulated curve reproduced the match-based psychometric function observed in human near/far discrimination. The dot density was 25% for both simulation and observation. We predicted that as the dot density increased, the performance for aRDSs should decrease below chance (i.e., reversed depth), and the level of anticorrelation that nullifies depth perception should also decrease. We suggest that cross-matching serves as a simple computation underlying the match-based disparity signals in stereoscopic depth perception.