Stereoscopic depth processing in the visual cortex: a coarse-to-fine mechanism

Prefrontal projections modulate recurrent circuitry in the insular cortex to support short-term memory

Short-term memory (STM) maintains information during a short delay period. How long-range and local connections interact to support STM encoding remains elusive. Here, we tackle the problem focusing on long-range projections from the medial prefrontal cortex (mPFC) to the anterior agranular insular cortex (aAIC) in head-fixed mice performing an olfactory delayed-response task. Optogenetic and electrophysiological experiments reveal the behavioral importance of the two regions in encoding STM information. Spike-correlogram analysis reveals strong local and cross-region functional coupling (FC) between memory neurons encoding the same information. Optogenetic suppression of mPFC-aAIC projections during the delay period reduces behavioral performance, the proportion of memory neurons, and memory-specific FC within the aAIC, whereas optogenetic excitation enhances all of them. mPFC-aAIC projections also bidirectionally modulate the efficacy of STM-information transfer, measured by the contribution of FC spiking pairs to the memory-coding ability of following neurons. Thus, prefrontal projections modulate insular neurons' functional connectivity and memory-coding ability to support STM.

Coarse-to-fine interaction on perceived depth in compound grating

To encode binocular disparity, the visual system uses a pair of left eye and right eye bandpass filters with either a position or a phase offset between them. Such pairs are considered to exit at multiple scales to encode a wide range of disparity. However, local disparity measurements by bandpass mechanisms can be ambiguous, particularly when the actual disparity is larger than a half-cycle of the preferred spatial frequency of the filter, which often occurs in fine scales. In this study, we investigated whether the visual system uses a coarse-to-fine interaction to resolve this ambiguity at finer scales for depth estimation from disparity. The stimuli were stereo grating patches composed of a target and comparison patterns. The target patterns contained spatial frequencies of 1 and 4 cycles per degree (cpd). The phase disparity of the low-frequency component was 0° (at the horopter), -90° (uncrossed), or 90° (crossed), and that of the high-frequency components was changed independent of the low-frequency disparity, in the range between -90° (uncrossed) and 90° (crossed). The observers' task was to indicate whether the target appeared closer to the comparison pattern, which always shared the disparity with the low-frequency component of the target. Regardless of whether the comparison pattern was a 1-cpd + 4-cpd compound or a 1-cpd simple grating, the perceived depth order of the target and the comparison varied in accordance with the phase disparity of the high-frequency component of the target. This effect occurred not only when the low-frequency component was at the horopter, but also when it contained a large disparity corresponding to one cycle of the high-frequency component (±90°). Our findings suggest a coarse-to-fine interaction in multiscale disparity processing, in which the depth interpretation of the high-frequency changes based on the disparity of the low-frequency component.

Stereopsis without correspondence

Stereopsis has traditionally been considered a complex visual ability, restricted to large-brained animals. The discovery in the 1980s that insects, too, have stereopsis, therefore, challenged theories of stereopsis. How can such simple brains see in three dimensions? A likely answer is that insect stereopsis has evolved to produce simple behaviour, such as orienting towards the closer of two objects or triggering a strike when prey comes within range. Scientific thinking about stereopsis has been unduly anthropomorphic, for example assuming that stereopsis must require binocular fusion or a solution of the stereo correspondence problem. In fact, useful behaviour can be produced with very basic stereoscopic algorithms which make no attempt to achieve fusion or correspondence, or to produce even a coarse map of depth across the visual field. This may explain why some aspects of insect stereopsis seem poorly designed from an engineering point of view: for example, paying no attention to whether interocular contrast or velocities match. Such algorithms demonstrably work well enough in practice for their species, and may prove useful in particular autonomous applications. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Coarse-to-fine processing drives the efficient coding of natural scenes in mouse visual cortex

The visual system processes sensory inputs sequentially, perceiving coarse information before fine details. Here we study the neural basis of coarse-to-fine processing and its computational benefits in natural vision. We find that primary visual cortical neurons in awake mice respond to natural scenes in a coarse-to-fine manner, primarily driven by individual neurons rapidly shifting their spatial frequency preference from low to high over a brief response period. This shift transforms the population response in a way that counteracts the statistical regularities of natural scenes, thereby reducing redundancy and generating a more efficient neural representation. The increase in representational efficiency does not occur in either dark-reared or anesthetized mice, which show significantly attenuated coarse-to-fine spatial processing. Collectively, these results illustrate that coarse-to-fine processing is state dependent, develops postnatally via visual experience, and provides a computational advantage by generating more efficient representations of the complex spatial statistics of ethologically relevant natural scenes.

Skyberg et al. show that the visual cortex of mice processes natural scenes in a coarse-to-fine manner, driven by individual neuron’s temporal dynamics. These response dynamics, which require visual experience to develop, reduce redundancy in the neural code and lead to more efficient representations of complex visual stimuli.

Adaptive Tuning Curve Widths Improve Sample Efficient Learning

Natural brains perform miraculously well in learning new tasks from a small number of samples, whereas sample efficient learning is still a major open problem in the field of machine learning. Here, we raise the question, how the neural coding scheme affects sample efficiency, and make first progress on this question by proposing and analyzing a learning algorithm that uses a simple reinforce-type plasticity mechanism and does not require any gradients to learn low dimensional mappings. It harnesses three bio-plausible mechanisms, namely, population codes with bell shaped tuning curves, continous attractor mechanisms and probabilistic synapses, to achieve sample efficient learning. We show both theoretically and by simulations that population codes with broadly tuned neurons lead to high sample efficiency, whereas codes with sharply tuned neurons account for high final precision. Moreover, a dynamic adaptation of the tuning width during learning gives rise to both, high sample efficiency and high final precision. We prove a sample efficiency guarantee for our algorithm that lies within a logarithmic factor from the information theoretical optimum. Our simulations show that for low dimensional mappings, our learning algorithm achieves comparable sample efficiency to multi-layer perceptrons trained by gradient descent, although it does not use any gradients. Furthermore, it achieves competitive sample efficiency in low dimensional reinforcement learning tasks. From a machine learning perspective, these findings may inspire novel approaches to improve sample efficiency. From a neuroscience perspective, these findings suggest sample efficiency as a yet unstudied functional role of adaptive tuning curve width.

Disparity level identification using the voxel-wise Gabor model of fMRI data

Perceiving disparities is the intuitive basis for our understanding of the physical world. Although many electrophysiology studies have revealed the disparity-tuning characteristics of the neurons in the visual areas of the macaque brain, neuron population responses to disparity processing have seldom been investigated. Many disparity studies using functional magnetic resonance imaging (fMRI) have revealed the disparity-selective visual areas in the human brain. However, it is unclear how to characterize neuron population disparity-tuning responses using fMRI technique. In the present study, we constructed three voxel-wise encoding Gabor models to predict the voxel responses to novel disparity levels and used a decoding method to identify the new disparity levels from population responses in the cortex. Among the three encoding models, the fine-coarse model (FCM) that used fine/coarse disparities to fit the voxel responses to disparities outperformed the single model and uncrossed-crossed model. Moreover, the FCM demonstrated high accuracy in predicting voxel responses in V3A complex and high accuracy in identifying novel disparities from responses in V3A complex. Our results suggest that the FCM can better characterize the voxel responses to disparities than the other two models and V3A complex is a critical visual area for representing disparity information.

Realization of real-time X-ray stereoscopic vision during interventional procedures

During interventional procedures, the deficiencies of nonstereoscopic vision increase the difficulty of identifying the anteroposterior direction and pathways of vessels. Therefore, achieving real-time stereoscopic vision during interventional procedures is meaningful. Pairs of X-ray images were captured with identical parameter settings, except for different rotation angles (represented as the α angle). The resulting images at these α angles were used as left-eye and right-eye views and were horizontally merged into single left-right 3D images. Virtual reality (VR) glasses were used for achieving stereo vision. Pairs of X-ray images from four angiographies with different α angles (1.8-3.4°) were merged into left-right 3D images. Observation with VR glasses can produce realistic stereo views of vascular anatomical structure. The results showed that the optimal α angles accepted by the brain for generating stereo vision were within a narrow range (approximately 1.4-4.1°). Subsequent tests showed that during transcatheter arterial chemoembolization, 3D X-ray stereoscopic images provided significantly improved spatial discrimination and convenience for identifying the supply vessels of a liver tumor and its anteroposterior direction compared with plain X-ray images (all P < 0.01). Real-time X-ray stereoscopic vision can be easily achieved via the straightforward method described herein and has the potential to benefit patients during interventional procedures.

Fundamental computational constraints on the time course of perception and action

A cognitive system faced with contingent events that cause rapid changes in sensory data may (i) incrementally incorporate new data into the ongoing perceptual and motor processing; or (ii) restart processing on each new event; or (iii) sample the data and hold onto the sample until its processing is complete, while disregarding any contingent changes. We offer a set of computational first-principles arguments for a hypothesis, according to which any system that contends with certain classes of perception and behavioral control tasks must include the sample-and-hold option (possibly alongside the other two, which may be useful in other tasks). This hypothesis has implications for understanding the dynamics of perception and action. In particular, a sample-and-hold channel necessarily processes sensory data on some kind of cycle (which does not imply precise periodicity). Further, being prepared to face the world at all times requires that the sampling that initiates each cycle be triggered by every significant action on part of the agent itself, such as saccades. We survey a range of evidence for the sample-and-hold functionality, touching upon diverse phenomena such as attentional blink and backward masking, the yoking of olfaction to respiration, thalamocortical interactions, and metastable brain dynamics in perception and consciousness.

Improving stereo matching algorithm with adaptive cross-scale cost aggregation

Human beings process stereoscopic correspondence across multiple purposes like robot navigation, automatic driving, and virtual or augmented reality. However, this bioinspiration is ignored by state-of-the-art dense stereo correspondence matching methods. Cost aggregation is one of the critical steps in the stereo matching method. In this article, we propose an optimized cross-scale cost aggregation scheme with coarse-to-fine strategy for stereo matching. This scheme implements cross-scale cost aggregation with the smoothness constraint on neighborhood cost, which essentially extends the idea of the inter-scale and intra-scale consistency constraints to increase the matching accuracy. The neighborhood costs are not only used in the intra-scale consistency to ensure that the regularized costs vary smoothly in an eight-connected neighbors region but also incorporated with inter-scale consistency to optimize the disparity estimation. Additionally, the improved method introduces an adaptive scheme in each scale with different aggregation methods. Finally, experimental results evaluated both on classic Middlebury and Middlebury 2014 data sets show that the proposed method performs better than the cross-scale cost aggregation. The whole stereo correspondence algorithm achieves competitive performance in terms of both matching accuracy and computational efficiency. An extensive comparison, including the KITTI benchmark, illustrates the better performance of the proposed method also.

Thresholds for sine-wave corrugations defined by binocular disparity in random dot stereograms: Factor analysis of individual differences reveals two stereoscopic mechanisms tuned for spatial frequency

Threshold functions for sinusoidal depth corrugations typically reach their minimum (highest sensitivity) at spatial frequencies of 0.2-0.4 cycles/degree (cpd), with lower thresholds for horizontal than vertical corrugations at low spatial frequencies. To elucidate spatial frequency and orientation tuning of stereoscopic mechanisms, we measured the disparity sensitivity functions, and used factor analytic techniques to estimate the existence of independent underlying stereo channels. The data set (N = 30 individuals) was for horizontal and vertical corrugations of spatial frequencies ranging from 0.1 to 1.6 cpd. A principal component analysis of disparity sensitivities (log-arcsec) revealed that two significant factors accounted for 70% of the variability. Following Varimax rotation to approximate "simple structure", one factor clearly loaded onto low spatial frequencies (≤0.4 cpd), and a second was tuned to higher spatial frequencies (≥0.8 cpd). Each factor had nearly identical tuning (loadings) for horizontal and vertical patterns. The finding of separate factors for low and high spatial frequencies is consistent with previous studies. The failure to find separate factors for horizontal and vertical corrugations is somewhat surprising because the neuronal mechanisms are believed to be different. Following an oblique rotation (Direct Oblimin), the two factors correlated significantly, suggesting some interdependence rather than full independence between the two factors.

Clinical features of acute acquired comitant esotropia in the Chinese populations

Acute acquired comitant esotropia (AACE) is an unusual presentation of esotropia that occurs after infancy. This study was aimed to study the clinical features and the differences between children and adult patients with AACE in the Chinese populations.This was a retrospective analysis of patients diagnosed with AACE over 4 years; 69 patients (25 females and 44 males) were identified. The patients were divided into 3 groups: < 10 year-old (n = 6, 8.7%), 10-18 year-old (n = 23, 33.3%), and ≥18 year-old (n = 40, 58.0%). Patients underwent medical history, brain and orbital computed tomography, and ophthalmological and orthoptic examinations.The refractions of AACE patients varied among age groups: patients < 10 year-old had mild hypermetropia, while older children and adults showed moderate-to-high myopia (P < .001). The mean angles of esotropia were significantly larger in young children compared with older children and adults (P = .005). There was no significant difference in binocularity detected by either synoptophore or TNO stereoscopic testing among different disease durations. Stereopsis detected by synoptophore and TNO testing showed no significant difference at duration within half a year, but the stereopsis measured by TNO was significantly worse than that detected by synoptophore with extending disease duration (P < .05).AACE seems to occur mostly in older children and adults in the Chinese population. Younger children with AACE seem to demonstrate a common trait of mild hypermetropic refractive errors, while myopia can be seen in older children and adult patients. The duration from onset to treatment of esotropia does not affect the preoperative binocularity.

Disparity processing in primary visual cortex

The first step in binocular stereopsis is to match features on the left retina with the correct features on the right retina, discarding 'false' matches. The physiological processing of these signals starts in the primary visual cortex, where the binocular energy model has been a powerful framework for understanding the underlying computation. For this reason, it is often used when thinking about how binocular matching might be performed beyond striate cortex. But this step depends critically on the accuracy of the model, and real V1 neurons show several properties that suggest they may be less sensitive to false matches than the energy model predicts. Several recent studies provide empirical support for an extended version of the energy model, in which the same principles are used, but the responses of single neurons are described as the sum of several subunits, each of which follows the principles of the energy model. These studies have significantly improved our understanding of the role played by striate cortex in the stereo correspondence problem.This article is part of the themed issue 'Vision in our three-dimensional world'.

Dynamic mechanisms of visually guided 3D motion tracking

The continuous perception of motion-through-depth is critical for both navigation and interacting with objects in a dynamic three-dimensional (3D) world. Here we used 3D tracking to simultaneously assess the perception of motion in all directions, facilitating comparisons of responses to motion-through-depth to frontoparallel motion. Observers manually tracked a stereoscopic target as it moved in a 3D Brownian random walk. We found that continuous tracking of motion-through-depth was selectively impaired, showing different spatiotemporal properties compared with frontoparallel motion tracking. Two separate factors were found to contribute to this selective impairment. The first is the geometric constraint that motion-through-depth yields much smaller retinal projections than frontoparallel motion, given the same object speed in the 3D environment. The second factor is the sluggish nature of disparity processing, which is present even for frontoparallel motion tracking of a disparity-defined stimulus. Thus, despite the ecological importance of reacting to approaching objects, both the geometry of 3D vision and the nature of disparity processing result in considerable impairments for tracking motion-through-depth using binocular cues.NEW & NOTEWORTHY We characterize motion perception continuously in all directions using an ecologically relevant, manual target tracking paradigm we recently developed. This approach reveals a selective impairment to the perception of motion-through-depth. Geometric considerations demonstrate that this impairment is not consistent with previously observed spatial deficits (e.g., stereomotion suppression). However, results from an examination of disparity processing are consistent with the longer latencies observed in discrete, trial-based measurements of the perception of motion-through-depth.

Cross-scale cost aggregation integrating intrascale smoothness constraint with weighted least squares in stereo matching

Cross-scale cost aggregation (CSCA) allows pixel-wise multiscale interaction in the aggregated cost computation. This kind of multiscale constraint strengthens the consistency of interscale cost volume and behaves well in a textureless region, compared with single-scale cost aggregation. However, the relationship between neighbors' cost is ignored. Based on the prior knowledge that costs should vary smoothly, except at object boundaries, the smoothness constraint on cost in a neighborhood system is integrated into the CSCA model with weighted least squares for reliable matching in this paper. Our improved algorithm not only has the advantage of CSCA in computational efficiency, but also performs better than CSCA, especially on the KITTI data sets. Experimental evidence demonstrates that the proposed algorithm outperforms CSCA in textureless and discontinuous regions. Quantitative evaluations demonstrate the effectiveness and efficiency of the proposed method for improving disparity estimation accuracy.

Sharpening coarse-to-fine stereo vision by perceptual learning: asymmetric transfer across the spatial frequency spectrum

Neurons in the early visual cortex are finely tuned to different low-level visual features, forming a multi-channel system analysing the visual image formed on the retina in a parallel manner. However, little is known about the potential 'cross-talk' among these channels. Here, we systematically investigated whether stereoacuity, over a large range of target spatial frequencies, can be enhanced by perceptual learning. Using narrow-band visual stimuli, we found that practice with coarse (low spatial frequency) targets substantially improves performance, and that the improvement spreads from coarse to fine (high spatial frequency) three-dimensional perception, generalizing broadly across untrained spatial frequencies and orientations. Notably, we observed an asymmetric transfer of learning across the spatial frequency spectrum. The bandwidth of transfer was broader when training was at a high spatial frequency than at a low spatial frequency. Stereoacuity training is most beneficial when trained with fine targets. This broad transfer of stereoacuity learning contrasts with the highly specific learning reported for other basic visual functions. We also revealed strategies to boost learning outcomes 'beyond-the-plateau'. Our investigations contribute to understanding the functional properties of the network subserving stereovision. The ability to generalize may provide a key principle for restoring impaired binocular vision in clinical situations.

Spatio-temporal properties of letter crowding

Crowding between adjacent letters has been investigated primarily as a spatial effect. The purpose of this study was to investigate the spatio-temporal properties of letter crowding. Specifically, we examined the systematic changes in the degradation effects in letter identification performance when adjacent letters were presented with a temporal asynchrony, as a function of letter separation and between the fovea and the periphery. We measured proportion-correct performance for identifying the middle target letter in strings of three lowercase letters at the fovea and 10° in the inferior visual field, for a range of center-to-center letter separations and a range of stimulus onset asynchronies (SOA) between the target and flanking letters (positive SOAs: target preceded flankers). As expected, the accuracy for identifying the target letters reduces with decreases in letter separation. This crowding effect shows a strong dependency on SOAs, such that crowding is maximal between 0 and ∼100 ms (depending on conditions) and diminishes for larger SOAs (positive or negative). Maximal crowding does not require the target and flanking letters to physically coexist for the entire presentation duration. Most importantly, crowding can be minimized even for closely spaced letters if there is a large temporal asynchrony between the target and flankers. The reliance of letter identification performance on SOAs and how it changes with letter separations imply that the crowding effect can be traded between space and time. Our findings are consistent with the notion that crowding should be considered as a spatio-temporal, and not simply a spatial, effect.

Relating functional connectivity in V1 neural circuits and 3D natural scenes using Boltzmann machines

Bayesian theory has provided a compelling conceptualization for perceptual inference in the brain. Central to Bayesian inference is the notion of statistical priors. To understand the neural mechanisms of Bayesian inference, we need to understand the neural representation of statistical regularities in the natural environment. In this paper, we investigated empirically how statistical regularities in natural 3D scenes are represented in the functional connectivity of disparity-tuned neurons in the primary visual cortex of primates. We applied a Boltzmann machine model to learn from 3D natural scenes, and found that the units in the model exhibited cooperative and competitive interactions, forming a "disparity association field", analogous to the contour association field. The cooperative and competitive interactions in the disparity association field are consistent with constraints of computational models for stereo matching. In addition, we simulated neurophysiological experiments on the model, and found the results to be consistent with neurophysiological data in terms of the functional connectivity measurements between disparity-tuned neurons in the macaque primary visual cortex. These findings demonstrate that there is a relationship between the functional connectivity observed in the visual cortex and the statistics of natural scenes. They also suggest that the Boltzmann machine can be a viable model for conceptualizing computations in the visual cortex and, as such, can be used to predict neural circuits in the visual cortex from natural scene statistics.

Integration of Multiple Spatial Frequency Channels in Disparity-Sensitive Neurons in the Primary Visual Cortex

For our vivid perception of a 3-D world, the stereoscopic function begins in our brain by detecting slight shifts of image features between the two eyes, called binocular disparity. The primary visual cortex is the first stage of this processing, and neurons there are tuned to a limited range of spatial frequencies (SFs). However, our visual world is generally highly complex, composed of numerous features at a variety of scales, thereby having broadband SF spectra. This means that binocular information signaled by individual neurons is highly incomplete, and combining information across multiple SF bands must be essential for the visual system to function in a robust and reliable manner. In this study, we investigated whether the integration of information from multiple SF channels begins in the cat primary visual cortex. We measured disparity-selective responses in the joint left-right SF domain using sequences of dichoptically flashed grating stimuli consisting of various combinations of SFs and phases. The obtained interaction map in the joint SF domain reflects the degree of integration across different SF channels. Our data are consistent with the idea that disparity information is combined from multiple SF channels in a substantial fraction of complex cells. Furthermore, for the majority of these neurons, the optimal disparity is matched across the SF bands. These results suggest that a highly specific SF integration process for disparity detection starts in the primary visual cortex.

SIGNIFICANCE STATEMENT

Our visual world is broadband, containing features with a wide range of object scales. On the other hand, single neurons in the primary visual cortex are narrow-band, being tuned narrowly for a specific scale. For robust visual perception, narrow-band information of single neurons must be integrated eventually at some stage. We have examined whether such an integration process begins in the primary visual cortex with respect to binocular processing. The results suggest that a subset of cells appear to combine binocular information across multiple scales. Furthermore, for the majority of these neurons, an optimal parameter of binocular tuning is matched across multiple scales, suggesting the presence of a highly specific neural integration mechanism.

Solving stereo transparency with an extended coarse-to-fine disparity energy model

Modeling stereo transparency with physiologically plausible mechanisms is challenging because in such frameworks, large receptive fields mix up overlapping disparities, whereas small receptive fields can reliably compute only small disparities. It seems necessary to combine information across scales. A coarse-to-fine disparity energy model, with both position- and phase-shift receptive fields, has already been proposed. However, because each scale decodes only one disparity for each location and uses the decoded disparity to select cells at the next scale, this model cannot represent overlapping surfaces at different depths. We have extended the model to solve stereo transparency. First, we introduce multiplicative connections from cells at one scale to the next to implement coarse-to-fine computation. The connection is the strongest when the presynaptic cell's preferred disparity matches the postsynaptic cell's position-shift parameter, encouraging the next scale to encode residual disparities with the more reliable phase-shift mechanism. This modification not only eliminates the artificial decoding and selection steps of the original model but also enables maintenance of complete population responses throughout the coarse-to-fine process. Second, because of this modification, explicit decoding is no longer necessary but rather is for visualization only. We use a simple threshold criterion to decode multiple disparities from population energy responses instead of a single disparity in the original model. We demonstrate our model using simulations on a variety of transparent and nontransparent stereograms. The model also reproduces psychophysically observed disparity interactions (averaging, thickening, attraction, and repulsion) as the depth separation between two overlapping planes varies.

Receptive fields of visual neurons: the early years

This paper traces the history of the visual receptive field (RF) from Hartline to Hubel and Wiesel. Hartline (1938, 1940) found that an isolated optic nerve fiber in the frog could be excited by light falling on a small circular area of the retina. He called this area the RF, using a term first introduced by Sherrington (1906) in the tactile domain. In 1953 Kuffler discovered the antagonistic center-surround organization of cat RFs, and Barlow, Fitzhugh, and Kuffler (1957) extended this work to stimulus size and state of adaptation. Shortly thereafter, Lettvin and colleagues (1959) in an iconic paper asked "what the frog's eye tells the frog's brain". Meanwhile, Jung and colleagues (1952-1973) searched for the perceptual correlates of neuronal responses, and Jung and Spillmann (1970) proposed the term perceptive field (PF) as a psychophysical correlate of the RF. The Westheimer function (1967) enabled psychophysical measurements of the PF center and surround in human and monkey, which correlated closely with the underlying RF organization. The sixties and seventies were marked by rapid progress in RF research. Hubel and Wiesel (1959-1974), recording from neurons in the visual cortex of the cat and monkey, found elongated RFs selective for the shape, orientation, and position of the stimulus, as well as for movement direction and ocularity. These findings prompted the emergence in visual psychophysics of the concept of feature detectors selective for lines, bars, and edges, and contributed to a model of the RF in terms of difference of Gaussians (DOG) and Fourier channels. The distinction between simple, complex, and hypercomplex neurons followed. Although RF size increases towards the peripheral retina, its cortical representation remains constant due to the reciprocal relationship with the cortical magnification factor (M). This constitutes a uniform yardstick for M-scaled stimuli across the retina. Developmental studies have shown that RF properties are not fixed. RFs possess their full response inventory already at birth, but require the interaction with appropriate stimuli within a critical time window for refinement and consolidation. Taken together these findings paved the way for a better understanding of how objective properties of the external world are encoded to become subjective properties of the subjective, perceptual world.

Cross-matching: a modified cross-correlation underlying threshold energy model and match-based depth perception

Three-dimensional visual perception requires correct matching of images projected to the left and right eyes. The matching process is faced with an ambiguity: part of one eye's image can be matched to multiple parts of the other eye's image. This stereo correspondence problem is complicated for random-dot stereograms (RDSs), because dots with an identical appearance produce numerous potential matches. Despite such complexity, human subjects can perceive a coherent depth structure. A coherent solution to the correspondence problem does not exist for anticorrelated RDSs (aRDSs), in which luminance contrast is reversed in one eye. Neurons in the visual cortex reduce disparity selectivity for aRDSs progressively along the visual processing hierarchy. A disparity-energy model followed by threshold nonlinearity (threshold energy model) can account for this reduction, providing a possible mechanism for the neural matching process. However, the essential computation underlying the threshold energy model is not clear. Here, we propose that a nonlinear modification of cross-correlation, which we term “cross-matching,” represents the essence of the threshold energy model. We placed half-wave rectification within the cross-correlation of the left-eye and right-eye images. The disparity tuning derived from cross-matching was attenuated for aRDSs. We simulated a psychometric curve as a function of graded anticorrelation (graded mixture of aRDS and normal RDS); this simulated curve reproduced the match-based psychometric function observed in human near/far discrimination. The dot density was 25% for both simulation and observation. We predicted that as the dot density increased, the performance for aRDSs should decrease below chance (i.e., reversed depth), and the level of anticorrelation that nullifies depth perception should also decrease. We suggest that cross-matching serves as a simple computation underlying the match-based disparity signals in stereoscopic depth perception.

Neural mechanisms of coarse-to-fine discrimination in the visual cortex

Vision is a dynamic process that refines the spatial scale of analysis over time, as evidenced by a progressive improvement in the ability to detect and discriminate finer details. To understand coarse-to-fine discrimination, we studied the dynamics of spatial frequency (SF) response using reverse correlation in the primary visual cortex (V1) of the primate. In a majority of V1 cells studied, preferred SF either increased monotonically with time (group 1) or changed nonmonotonically, with an initial increase followed by a decrease (group 2). Monotonic shift in preferred SF occurred with or without an early suppression at low SFs. Late suppression at high SFs always accompanied nonmonotonic SF dynamics. Bayesian analysis showed that SF discrimination performance and best discriminable SF frequencies changed with time in different ways in the two groups of neurons. In group 1 neurons, SF discrimination performance peaked on both left and right flanks of the SF tuning curve at about the same time. In group 2 neurons, peak discrimination occurred on the right flank (high SFs) later than on the left flank (low SFs). Group 2 neurons were also better discriminators of high SFs. We examined the relationship between the time at which SF discrimination performance peaked on either flank of the SF tuning curve and the corresponding best discriminable SFs in both neuronal groups. This analysis showed that the population best discriminable SF increased with time in V1. These results suggest neural mechanisms for coarse-to-fine discrimination behavior and that this process originates in V1 or earlier.

Perceiving and acting in depth in Williams syndrome and typical development

Individuals with the neurodevelopmental disorder Williams syndrome (WS) often report difficulty processing and acting in depth, such as crossing roads or reaching for objects; however little research attention has been directed at understanding depth perception and action in depth in WS and whether deficits in depth perception have an ocular or perceptual root in this group. This study assessed the extent and relationship of deficits in stereopsis (binocular, three dimensional vision) and actions performed in depth in WS, as well as in typically developing participants (TD) matched for non-verbal ability. Stereoacuity was age-appropriate in the TD group but at the level of a TD three year old in WS; one third of the WS group did not show evidence of stereopsis. When monocularly acting in depth there was no difference between the WS and TD groups. When binocularly acting in depth the WS group that did not exhibit stereopsis were significantly poorer than the TD group and the WS group that exhibited stereopsis. When assessing the relationship between stereoacuity and action in depth, stereoacuity negatively correlated with binocular action in depth for the WS group with stereopsis, but not the TD group. Therefore, no deficits in monocular depth perception in WS were evidenced, yet significant deficits are exhibited in binocular depth perception and action. Importantly action in depth under binocular viewing may be a useful gross screening measure for stereodeficits in WS. Remediation of depth perception deficits in WS could train further understanding of monocular cues to compensate for poor stereopsis.

Selective stimulation of neurons in visual cortex enables segregation of slow and fast connections

Organization of the central visual pathway is generally studied from a perspective of feedforward processes. However, there are horizontal connections and also strong feedback from extra striate to visual cortex. Here, we use visual stimuli designed to maximize relative differential involvements of these three main types of connections. The approach relies on differences between stimulation within the classical receptive field (CRF) and that of the surround region. Although previous studies have used similar approaches, they were limited primarily to spatial segregation of neural connections. Our experimental design provides clear segregation of fast and slow components of surround modulation. We assume these are mediated by feedback and horizontal connections, respectively, but other factors may be involved. Our results imply that both horizontal and feedback connections contribute to integration of visual information outside the CRF and provide suppressive or facilitative modulation. For a given cell, modulation may change in strength and sign from suppression to facilitation or the reverse depending on surround parameters. Sub-threshold input from the CRF surround increases local field potential (LFP) power in distinct frequency ranges which differ for suppression and facilitation. Horizontal connections have delayed CRF-surround modulation and are sensitive to position changes in the surround. Therefore, surround information beyond the CRF is initially processed by fast connections which we consider to be feedback, whereas spatially tuned mechanisms are relatively slow and presumably mediated by horizontal connections. Overall, results suggest that convergent fast (feedforward) inputs determine size and structure of the CRFs of recipient cells in visual cortex. And fast connections from extra striate regions (feedback) plus slow-tuned connections (horizontal) within visual cortex contribute to spatial influences of CRF surround activation.

Sample skewness as a statistical measurement of neuronal tuning sharpness

We propose using the statistical measurement of the sample skewness of the distribution of mean firing rates of a tuning curve to quantify sharpness of tuning. For some features, like binocular disparity, tuning curves are best described by relatively complex and sometimes diverse functions, making it difficult to quantify sharpness with a single function and parameter. Skewness provides a robust nonparametric measure of tuning curve sharpness that is invariant with respect to the mean and variance of the tuning curve and is straightforward to apply to a wide range of tuning, including simple orientation tuning curves and complex object tuning curves that often cannot even be described parametrically. Because skewness does not depend on a specific model or function of tuning, it is especially appealing to cases of sharpening where recurrent interactions among neurons produce sharper tuning curves that deviate in a complex manner from the feedforward function of tuning. Since tuning curves for all neurons are not typically well described by a single parametric function, this model independence additionally allows skewness to be applied to all recorded neurons, maximizing the statistical power of a set of data. We also compare skewness with other nonparametric measures of tuning curve sharpness and selectivity. Compared to these other nonparametric measures tested, skewness is best used for capturing the sharpness of multimodal tuning curves defined by narrow peaks (maximum) and broad valleys (minima). Finally, we provide a more formal definition of sharpness using a shape-based information gain measure and derive and show that skewness is correlated with this definition.

Delayed suppression shapes disparity selective responses in monkey V1

The stereo correspondence problem poses a challenge to visual neurons because localized receptive fields potentially cause false responses. Neurons in the primary visual cortex (V1) partially resolve this problem by combining excitatory and suppressive responses to encode binocular disparity. We explored the time course of this combination in awake, monkey V1 neurons using subspace mapping of receptive fields. The stimulus was a binocular noise pattern constructed from discrete spatial frequency components. We forward correlated the firing of the V1 neuron with the occurrence of binocular presentations of each spatial frequency component. The forward correlation yielded a complete set of response time courses to every combination of spatial frequency and interocular phase difference. Some combinations produced suppressive responses. Typically, if an interocular phase difference for a given spatial frequency produced strong excitation, we saw suppression in response to the opposite interocular phase difference at lower spatial frequencies. The suppression was delayed relative to the excitation, with a median difference in latency of 7 ms. We found that the suppressive mechanism explains a well-known mismatch of monocular and binocular signals. The suppressive components increased power at low spatial frequencies in disparity tuning, whereas they reduced the monocular response to low spatial frequencies. This long-recognized mismatch of binocular and monocular signals reflects a suppressive mechanism that helps reduce the response to false matches.

Visual perception: one world from two eyes

Binocular vision requires us to match up the different views of the world seen by each eye. Computational models of primary visual cortex describe how the brain begins this process. Recurrent connections help suppress the response to false matches.

Joint tuning for direction of motion and binocular disparity in macaque MT is largely separable

Neurons in sensory cortical areas are tuned to multiple dimensions, or features, of their sensory space. Understanding how single neurons represent multiple features is of great interest for determining the informative dimensions of the neurons' response, the decoding algorithms appropriate for extracting this information from the neuronal population, and for determining where specific transformations occur along the visual hierarchy. Despite the established role of cortical area MT in judgments of motion and depth, it is not known how individual neurons jointly encode the two dimensions. We investigated the joint tuning of individual MT neurons for two visual features: direction of motion and binocular disparity, an important depth cue. We found that a separable, multiplicative combination of tuning for the two features can account for more than 90% of the variance in the joint tuning function for over 91% of MT neurons. These results suggest 1) that each feature can be read out independently from MT by simply averaging across the population without regard to the other feature and 2) that the inseparable representations seen in subsequent areas, such as MST, must be computed beyond MT. Intriguingly, we found that the remaining nonseparable component of the joint tuning function often manifested as small but systematic changes in the neurons' preferences for one feature as the other one was varied. We believe this reflects the local columnar organization of tuning for direction and binocular disparity in MT, indicating that joint tuning may provide a new tool with which to probe functional architecture.

Toward an impairment metric for stereoscopic video: a full-reference video quality metric to assess compressed stereoscopic video

The quality assessment of impaired stereoscopic video is a key element in designing and deploying advanced immersive media distribution platforms. A widely accepted quality metric to measure impairments of stereoscopic video is, however, still to be developed. As a step toward finding a solution to this problem, this paper proposes a full reference stereoscopic video quality metric to measure the perceptual quality of compressed stereoscopic video. A comprehensive set of subjective experiments is performed with 14 different stereoscopic video sequences, which are encoded using both the H.264 and high efficiency video coding compliant video codecs, to develop a subjective test results database of 116 test stimuli. The subjective results are analyzed using statistical techniques to uncover different patterns of subjective scoring for symmetrically and asymmetrically encoded stereoscopic video. The subjective result database is subsequently used for training and validating a simple but effective stereoscopic video quality metric considering heuristics of binocular vision. The proposed metric performs significantly better than state-of-the-art stereoscopic image and video quality metrics in predicting the subjective scores. The proposed metric and the subjective result database will be made publicly available, and it is expected that the proposed metric and the subjective assessments will have important uses in advanced 3D media delivery systems.

Temporally advanced dynamic change of receptive field of lateral geniculate neurons during brief visual stimulation: Effects of brainstem peribrachial stimulation

Processing of visual information in the brain seems to proceed from initial fast but coarse to subsequent detailed processing. Such coarse-to-fine changes appear also in the response of single neurons in the visual pathway. In the dorsal lateral geniculate nucleus (dLGN), there is a dynamic change in the receptive field (RF) properties of neurons during visual stimulation. During a stimulus flash centered on the RF, the width of the RF-center, presumably related to spatial resolution, changes rapidly from large to small in an initial transient response component. In a subsequent sustained component, the RF-center width is rather stable apart from an initial slight widening. Several brainstem nuclei modulate the geniculocortical transmission in a state-dependent manner. Thus, modulatory input from cholinergic neurons in the peribrachial brainstem region (PBR) enhances the geniculocortical transmission during arousal. We studied whether such input also influences the dynamic RF-changes during visual stimulation. We compared dynamic changes of RF-center width of dLGN neurons during brief stimulus presentation in a control condition, with changes during combined presentation of the visual stimulus and electrical PBR-stimulation. The major finding was that PBR-stimulation gave an advancement of the dynamic change of the RF-center width such that the different response components occurred earlier. Consistent with previous studies, we also found that PBR-stimulation increased the gain of firing rate during the sustained response component. However, this increase of gain was particularly strong in the transition from the transient to the sustained component at the time when the center width was minimal. The results suggest that increased modulatory PBR-input not only increase the gain of the geniculocortical transmission, but also contributes to faster dynamics of transmission. We discuss implications for possible effects on visual spatial resolution.

Short-time scale dynamics in the responses to multiple stimuli in visual cortex

Many previous studies have used the presentation of multiple stimuli in the receptive fields (RFs) of visual cortical neurons to explore how neurons might operate on multiple inputs. Most of these experiments have used two fixed stimulus locations within the RF of each neuron. Here the effects of using different positions within the RF of a neuron were explored. The stimuli were presented singly at one of six locations, and also at 15 pair-wise combinations, for 24 V2 cortical neurons in two macaque monkeys. There was considerable variability in how pairs of stimuli interacted within the receptive field of any given neuron: changing the position of the stimuli could result in enhancement, winner-take-all, or suppression relative to the strongest response to a stimulus presented by itself. Across the population of neurons there was no correlation between response strength and response latency. However, for many stimulus pairs the response latency was tightly locked to the shortest response latency of any single stimulus presented by itself independent of changes in response magnitude. In other words, a stimulus that by itself elicited a relatively long latency response, would often affect the magnitude of the response to a pair of stimuli, but not change the latency. These results may provide constraints on the development of models of cortical information processing.

Robust multiscale stereo matching from fundus images with radiometric differences

A robust multiscale stereo matching algorithm is proposed to find reliable correspondences between low contrast and weakly textured retinal image pairs with radiometric differences. Existing algorithms designed to deal with piecewise planar surfaces with distinct features and Lambertian reflectance do not apply in applications such as 3D reconstruction of medical images including stereo retinal images. In this paper, robust pixel feature vectors are formulated to extract discriminative features in the presence of noise in scale space, through which the response of low-frequency mechanisms alter and interact with the response of high-frequency mechanisms. The deep structures of the scene are represented with the evolution of disparity estimates in scale space, which distributes the matching ambiguity along the scale dimension to obtain globally coherent reconstructions. The performance is verified both qualitatively by face validity and quantitatively on our collection of stereo fundus image sets with ground truth, which have been made publicly available as an extension of standard test images for performance evaluation.

Coarse-to-fine changes of receptive fields in lateral geniculate nucleus have a transient and a sustained component that depend on distinct mechanisms

Visual processing in the brain seems to provide fast but coarse information before information about fine details. Such dynamics occur also in single neurons at several levels of the visual system. In the dorsal lateral geniculate nucleus (LGN), neurons have a receptive field (RF) with antagonistic center-surround organization, and temporal changes in center-surround organization are generally assumed to be due to a time-lag of the surround activity relative to center activity. Spatial resolution may be measured as the inverse of center size, and in LGN neurons RF-center width changes during static stimulation with durations in the range of normal fixation periods (250-500 ms) between saccadic eye-movements. The RF-center is initially large, but rapidly shrinks during the first ~100 ms to a rather sustained size. We studied such dynamics in anesthetized cats during presentation (250 ms) of static spots centered on the RF with main focus on the transition from the first transient and highly dynamic component to the second more sustained component. The results suggest that the two components depend on different neuronal mechanisms that operate in parallel and with partial temporal overlap rather than on a continuously changing center-surround balance. Results from mathematical modeling further supported this conclusion. We found that existing models for the spatiotemporal RF of LGN neurons failed to account for our experimental results. The modeling demonstrated that a new model, in which the response is given by a sum of an early transient component and a partially overlapping sustained component, adequately accounts for our experimental data.

Coarse to fine dynamics of monocular and binocular processing in human pattern vision

Biological image processing has been hypothesized to adopt a coarse to fine strategy: the image is initially analyzed at a coarse spatial scale, and this analysis is then used to guide subsequent inspection at a finer scale. Neurons in visual cortex often display response characteristics that are consistent with this hypothesis for both monocular and binocular signals. Puzzlingly, measurements in human observers have failed to expose similar coarse to fine dynamics for human pattern vision, questioning the applicability of direct parallels between single neurons and perception. We performed a series of measurements using experimental protocols that were specifically designed to examine this question in more detail. We were able to confirm that, when the analysis is restricted to the linear properties of the perceptual process, no coarse to fine dynamics were evident in the data. However, when the analysis was extended to nonlinear descriptors, a clear coarse to fine structure emerged that consisted of two processes: an early nonlinear process operating on a coarse spatial scale followed by a linear process operating on a fine spatial scale. These results potentially serve to reduce the gap between the electrophysiological and behavioral findings.

Binocular vision

This essay reviews major developments - empirical and theoretical - in the field of binocular vision during the last 25years. We limit our survey primarily to work on human stereopsis, binocular rivalry and binocular contrast summation, with discussion where relevant of single-unit neurophysiology and human brain imaging. We identify several key controversies that have stimulated important work on these problems. In the case of stereopsis those controversies include position vs. phase encoding of disparity, dependence of disparity limits on spatial scale, role of occlusion in binocular depth and surface perception, and motion in 3D. In the case of binocular rivalry, controversies include eye vs. stimulus rivalry, role of "top-down" influences on rivalry dynamics, and the interaction of binocular rivalry and stereopsis. Concerning binocular contrast summation, the essay focuses on two representative models that highlight the evolving complexity in this field of study.

Triphasic dynamics of stimulus-dependent information flow between single neurons in macaque inferior temporal cortex

The functional connectivity between cortical neurons is not static and is known to exhibit contextual modulations in terms of the coupling strength. Here we hypothesized that the information flow in a cortical local circuit exhibits complex forward-and-back dynamics, and conducted Granger causality analysis between the neuronal spike trains that were simultaneously recorded from macaque inferior temporal (IT) cortex while the animals performed a visual object discrimination task. Spikes from neuron pairs with a displaced peak on the cross-correlogram (CCG) showed Granger causality in the gamma-frequency range (30-80 Hz) with the dominance in the direction consistent with the CCG peak (forward direction). Although, in a classical view, the displaced CCG peak has been interpreted as an indicative of a pauci-synaptic serial linkage, temporal dynamics of the gamma Granger causality after stimulus onset exhibited a more complex triphasic pattern, with a transient forward component followed by a slowly developing backward component and subsequent reappearance of the forward component. These triphasic dynamics of causality were not explained by the firing rate dynamics and were not observed for cell pairs that exhibited a center peak on the CCG. Furthermore, temporal dynamics of Granger causality depended on the feature configuration within the presented object. Together, these results demonstrate that the classical view of functional connectivity could be expanded to incorporate more complex forward-and-back dynamics and also imply that multistage processing in the recognition of visual objects might be implemented by multiphasic dynamics of directional information flow between single neurons in a local circuit in the IT cortex.

Modeling stereopsis via Markov random field

Markov random field (MRF) and belief propagation have given birth to stereo vision algorithms with top performance. This article explores their biological plausibility. First, an MRF model guided by physiological and psychophysical facts was designed. Typically an MRF-based stereo vision algorithm employs a likelihood function that reflects the local similarity of two regions and a potential function that models the continuity constraint. In our model, the likelihood function is constructed on the basis of the disparity energy model because complex cells are considered as front-end disparity encoders in the visual pathway. Our likelihood function is also relevant to several psychological findings. The potential function in our model is constrained by the psychological finding that the strength of the cooperative interaction minimizing relative disparity decreases as the separation between stimuli increases. Our model is tested on three kinds of stereo images. In simulations on images with repetitive patterns, we demonstrate that our model could account for the human depth percepts that were previously explained by the second-order mechanism. In simulations on random dot stereograms and natural scene images, we demonstrate that false matches introduced by the disparity energy model can be reliably removed using our model. A comparison with the coarse-to-fine model shows that our model is able to compute the absolute disparity of small objects with larger relative disparity. We also relate our model to several physiological findings. The hypothesized neurons of the model are selective for absolute disparity and have facilitative extra receptive field. There are plenty of such neurons in the visual cortex. In conclusion, we think that stereopsis can be implemented by neural networks resembling MRF.

Cooperative and competitive interactions facilitate stereo computations in macaque primary visual cortex

Inferring depth from binocular disparities is a difficult problem for the visual system because local features in the left- and right-eye images must be matched correctly to solve this "stereo correspondence problem." Cortical architecture and computational studies suggest that lateral interactions among neurons could help resolve local uncertainty about disparity encoded in individual neurons by incorporating contextual constraints. We found that correlated activity among pairs of neurons in primary visual cortex depended both on disparity-tuning relationships and the stimuli displayed within the receptive fields of the neurons. Nearby pairs of neurons with distinct disparity tuning exhibited a decrease in spike correlation at competing disparities soon after response onset. Distant neuronal pairs of similar disparity tuning exhibited an increase in spike correlation at mutually preferred disparities. The observed correlated activity and response dynamics suggests that local competitive and distant cooperative interactions improve disparity tuning of individual neurons over time. Such interactions could represent a neural substrate for the principal constraints underlying cooperative stereo algorithms.