Transformation from a retinal to a cyclopean representation in human visual cortex

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues - a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.

A binocular synaptic network supports interocular response alignment in visual cortical neurons

In visual cortex, signals from the two eyes merge to form a coherent binocular representation. Here we investigate the synaptic interactions underlying the binocular representation of stimulus orientation in ferret visual cortex with in vivo calcium imaging of layer 2/3 neurons and their dendritic spines. Individual neurons with aligned somatic responses received a mixture of monocular and binocular synaptic inputs. Surprisingly, monocular pathways alone could not account for somatic alignment because ipsilateral monocular inputs poorly matched somatic preference. Binocular inputs exhibited different degrees of interocular alignment, and those with a high degree of alignment (congruent) had greater selectivity and somatic specificity. While congruent inputs were similar to others in measures of strength, simulations show that the number of active congruent inputs predicts aligned somatic output. Our study suggests that coherent binocular responses derive from connectivity biases that support functional amplification of aligned signals within a heterogeneous binocular intracortical network.

Human primary visual cortex shows larger population receptive fields for binocular disparity-defined stimuli

The visual perception of 3D depth is underpinned by the brain's ability to combine signals from the left and right eyes to produce a neural representation of binocular disparity for perception and behaviour. Electrophysiological studies of binocular disparity over the past 2 decades have investigated the computational role of neurons in area V1 for binocular combination, while more recent neuroimaging investigations have focused on identifying specific roles for different extrastriate visual areas in depth perception. Here we investigate the population receptive field properties of neural responses to binocular information in striate and extrastriate cortical visual areas using ultra-high field fMRI. We measured BOLD fMRI responses while participants viewed retinotopic mapping stimuli defined by different visual properties: contrast, luminance, motion, correlated and anti-correlated stereoscopic disparity. By fitting each condition with a population receptive field model, we compared quantitatively the size of the population receptive field for disparity-specific stimulation. We found larger population receptive fields for disparity compared with contrast and luminance in area V1, the first stage of binocular combination, which likely reflects the binocular integration zone, an interpretation supported by modelling of the binocular energy model. A similar pattern was found in region LOC, where it may reflect the role of disparity as a cue for 3D shape. These findings provide insight into the binocular receptive field properties underlying processing for human stereoscopic vision.

Evidence for strictly monocular processing in visual motion opponency and Glass pattern perception

When presented with locally paired dots moving in opposite directions, motion selective neurons in the middle temporal cortex (MT) reduce firing while neurons in V1 are unaffected. This physiological effect is known as motion opponency. The current study used psychophysics to investigate the neural circuit underlying motion opponency. We asked whether opposing motion signals could arrive from different eyes into the receptive field of a binocular neuron while still maintaining motion opponency. We took advantage of prior findings that orientation discrimination of the motion axis (along which paired dots oscillate) is harder when dots move counter-phase than in-phase, an effect associated with motion opponency. We found that such an effect disappeared when paired dots originated from different eyes. This suggests that motion opponency, at some point, involves strictly monocular processing. This does not mean that motion opponency is entirely monocular. Further, we found that the effect of a Glass pattern disappeared under similar viewing conditions, suggesting that Glass pattern perception also involves some strictly monocular processing.

Multivariate Analysis of BOLD Activation Patterns Recovers Graded Depth Representations in Human Visual and Parietal Cortex

Navigating through natural environments requires localizing objects along three distinct spatial axes. Information about position along the horizontal and vertical axes is available from an object’s position on the retina, while position along the depth axis must be inferred based on second-order cues such as the disparity between the images cast on the two retinae. Past work has revealed that object position in two-dimensional (2D) retinotopic space is robustly represented in visual cortex and can be robustly predicted using a multivariate encoding model, in which an explicit axis is modeled for each spatial dimension. However, no study to date has used an encoding model to estimate a representation of stimulus position in depth. Here, we recorded BOLD fMRI while human subjects viewed a stereoscopic random-dot sphere at various positions along the depth (z) and the horizontal (x) axes, and the stimuli were presented across a wider range of disparities (out to ∼40 arcmin) compared to previous neuroimaging studies. In addition to performing decoding analyses for comparison to previous work, we built encoding models for depth position and for horizontal position, allowing us to directly compare encoding between these dimensions. Our results validate this method of recovering depth representations from retinotopic cortex. Furthermore, we find convergent evidence that depth is encoded most strongly in dorsal area V3A.

How Ocular Dominance and Binocularity Are Reflected by the Population Receptive Field Properties

Purpose

The neural substrate of binocularity and sighting ocular dominance in humans is not clear. By utilizing the population receptive field (pRF) modeling technique, we explored whether these phenomena are associated with amplitude and pRF size differences.

Methods

The visual field maps of 13 subjects were scanned (3-T Skyra) while viewing drifting bar stimuli. Both eyes (binocular condition), the dominant eye and the nondominant eye (two monocular conditions) were stimulated in separate sessions. For each condition, pRF size and amplitude were assessed. Binocular summation ratios were calculated by dividing binocular by mean monocular values (amplitude and pRF size).

Results

No differences in pRF size were seen between the viewing conditions within each region, that is, either between binocular and monocular or between dominant and nondominant viewing conditions. Binocular amplitudes were higher than the monocular amplitudes, but similar among the dominant and nondominant eyes. Binocular summation ratios derived from amplitudes were significantly higher than one (∼1.2), while those ratios derived from pRF size were not. These effects were found in all studied areas along the visual hierarchy, starting in V1.

Conclusions

Neither the amplitude nor the pRF size show intereye difference and therefore cannot explain the different roles of the dominant and the nondominant eyes. Binocular, as compared to monocular vision, resulted in higher amplitudes, while receptive fields' sizes were similar, suggesting increased binocular response intensity as the basis for the binocular summation phenomenon. Our results could be applicable in imaging studies of monocular disease and studies that deal with nondisparity binocularity effects.

Binocular system with asymmetric eyes

I elaborate binocular geometry with a novel eye model that incorporates the fovea's temporalward displacement and the cornea and the lens' misalignment. The formulated binocular correspondence results in longitudinal horopters that are conic sections resembling empirical horopters. When the eye model's asymmetry parameters' range is that which is observed in healthy eyes, abathic distance also falls within its experimentally observed range. This range in abathic distance is similar to that of the vergence resting position distance. Further, the conic's orientation is specified by the eyes' version angle, integrating binocular geometry with eye movement. This integration presents the possibility for modeling 3D perceptual stability during physiological eye movements.

Binocular Mechanisms of 3D Motion Processing

The visual system must recover important properties of the external environment if its host is to survive. Because the retinae are effectively two-dimensional but the world is three-dimensional (3D), the patterns of stimulation both within and across the eyes must be used to infer the distal stimulus-the environment-in all three dimensions. Moreover, animals and elements in the environment move, which means the input contains rich temporal information. Here, in addition to reviewing the literature, we discuss how and why prior work has focused on purported isolated systems (e.g., stereopsis) or cues (e.g., horizontal disparity) that do not necessarily map elegantly on to the computations and complex patterns of stimulation that arise when visual systems operate within the real world. We thus also introduce the binoptic flow field (BFF) as a description of the 3D motion information available in realistic environments, which can foster the use of ecologically valid yet well-controlled stimuli. Further, it can help clarify how future studies can more directly focus on the computations and stimulus properties the visual system might use to support perception and behavior in a dynamic 3D world.

How to Build a Dichoptic Presentation System That Includes an Eye Tracker

The presentation of different stimuli to the two eyes, dichoptic presentation, is essential for studies involving 3D vision and interocular suppression. There is a growing literature on the unique experimental value of pupillary and oculomotor measures, especially for research on interocular suppression. Although obtaining eye-tracking measures would thus benefit studies that use dichoptic presentation, the hardware essential for dichoptic presentation (e.g. mirrors) often interferes with high-quality eye tracking, especially when using a video-based eye tracker. We recently described an experimental setup that combines a standard dichoptic presentation system with an infrared eye tracker by using infrared-transparent mirrors1. The setup is compatible with standard monitors and eye trackers, easy to implement, and affordable (on the order of US$1,000). Relative to existing methods it has the benefits of not requiring special equipment and posing few limits on the nature and quality of the visual stimulus. Here we provide a visual guide to the construction and use of our setup.

2D location biases depth-from-disparity judgments but not vice versa

Visual cognition in our 3D world requires understanding how we accurately localize objects in 2D and depth, and what influence both types of location information have on visual processing. Spatial location is known to play a special role in visual processing, but most of these findings have focused on the special role of 2D location. One such phenomena is the spatial congruency bias (Golomb, Kupitz, & Thiemann, 2014), where 2D location biases judgments of object features but features do not bias location judgments. This paradigm has recently been used to compare different types of location information in terms of how much they bias different types of features. Here we used this paradigm to ask a related question: whether 2D and depth-from-disparity location bias localization judgments for each other. We found that presenting two objects in the same 2D location biased position-in-depth judgments, but presenting two objects at the same depth (disparity) did not bias 2D location judgments. We conclude that an object's 2D location may be automatically incorporated into perception of its depth location, but not vice versa, which is consistent with a fundamentally special role for 2D location in visual processing.

Sensitivity to an Illusion of Sound Location in Human Auditory Cortex

Human listeners place greater weight on the beginning of a sound compared to the middle or end when determining sound location, creating an auditory illusion known as the Franssen effect. Here, we exploited that effect to test whether human auditory cortex (AC) represents the physical vs. perceived spatial features of a sound. We used functional magnetic resonance imaging (fMRI) to measure AC responses to sounds that varied in perceived location due to interaural level differences (ILD) applied to sound onsets or to the full sound duration. Analysis of hemodynamic responses in AC revealed sensitivity to ILD in both full-cue (veridical) and onset-only (illusory) lateralized stimuli. Classification analysis revealed regional differences in the sensitivity to onset-only ILDs, where better classification was observed in posterior compared to primary AC. That is, restricting the ILD to sound onset—which alters the physical but not the perceptual nature of the spatial cue—did not eliminate cortical sensitivity to that cue. These results suggest that perceptual representations of auditory space emerge or are refined in higher-order AC regions, supporting the stable perception of auditory space in noisy or reverberant environments and forming the basis of illusions such as the Franssen effect.

Differential patterns of 2D location versus depth decoding along the visual hierarchy

Visual information is initially represented as 2D images on the retina, but our brains are able to transform this input to perceive our rich 3D environment. While many studies have explored 2D spatial representations or depth perception in isolation, it remains unknown if or how these processes interact in human visual cortex. Here we used functional MRI and multi-voxel pattern analysis to investigate the relationship between 2D location and position-in-depth information. We stimulated different 3D locations in a blocked design: each location was defined by horizontal, vertical, and depth position. Participants remained fixated at the center of the screen while passively viewing the peripheral stimuli with red/green anaglyph glasses. Our results revealed a widespread, systematic transition throughout visual cortex. As expected, 2D location information (horizontal and vertical) could be strongly decoded in early visual areas, with reduced decoding higher along the visual hierarchy, consistent with known changes in receptive field sizes. Critically, we found that the decoding of position-in-depth information tracked inversely with the 2D location pattern, with the magnitude of depth decoding gradually increasing from intermediate to higher visual and category regions. Representations of 2D location information became increasingly location-tolerant in later areas, where depth information was also tolerant to changes in 2D location. We propose that spatial representations gradually transition from 2D-dominant to balanced 3D (2D and depth) along the visual hierarchy.

Feature-location binding in 3D: Feature judgments are biased by 2D location but not position-in-depth

A fundamental aspect of human visual perception is the ability to recognize and locate objects in the environment. Importantly, our environment is predominantly three-dimensional (3D), but while there is considerable research exploring the binding of object features and location, it is unknown how depth information interacts with features in the object binding process. A recent paradigm called the spatial congruency bias demonstrated that 2D location is fundamentally bound to object features, such that irrelevant location information biases judgments of object features, but irrelevant feature information does not bias judgments of location or other features. Here, using the spatial congruency bias paradigm, we asked whether depth is processed as another type of location, or more like other features. We initially found that depth cued by binocular disparity biased judgments of object color. However, this result seemed to be driven more by the disparity differences than the depth percept: Depth cued by occlusion and size did not bias color judgments, whereas vertical disparity information (with no depth percept) did bias color judgments. Our results suggest that despite the 3D nature of our visual environment, only 2D location information - not position-in-depth - seems to be automatically bound to object features, with depth information processed more similarly to other features than to 2D location.

Which way and how far? Tracking of translation and rotation information for human path integration

Path integration, the constant updating of the navigator's knowledge of position and orientation during movement, requires both visuospatial knowledge and memory. This study aimed to develop a systems-level understanding of human path integration by examining the basic building blocks of path integration in humans. To achieve this goal, we used functional imaging to examine the neural mechanisms that support the tracking and memory of translational and rotational components of human path integration. Critically, and in contrast to previous studies, we examined movement in translation and rotation tasks with no defined end-point or goal. Navigators accumulated translational and rotational information during virtual self-motion. Activity in hippocampus, retrosplenial cortex (RSC), and parahippocampal cortex (PHC) increased during both translation and rotation encoding, suggesting that these regions track self-motion information during path integration. These results address current questions regarding distance coding in the human brain. By implementing a modified delayed match to sample paradigm, we also examined the encoding and maintenance of path integration signals in working memory. Hippocampus, PHC, and RSC were recruited during successful encoding and maintenance of path integration information, with RSC selective for tasks that required processing heading rotation changes. These data indicate distinct working memory mechanisms for translation and rotation, which are essential for updating neural representations of current location. The results provide evidence that hippocampus, PHC, and RSC flexibly track task-relevant translation and rotation signals for path integration and could form the hub of a more distributed network supporting spatial navigation. Hum Brain Mapp 37:3636-3655, 2016. © 2016 Wiley Periodicals, Inc.

There and Back Again: Hippocampus and Retrosplenial Cortex Track Homing Distance during Human Path Integration

Path integration, the updating of position and orientation during movement, often involves tracking a home location. Here, we examine processes that could contribute to successful location tracking in humans. In particular, we investigate a homing vector model of path integration, whereby a navigator continuously tracks a trajectory back to the home location. To examine this model, we developed a loop task for fMRI, in which participants viewed movement that circled back to a home location in a sparse virtual environment. In support of a homing vector system, hippocampus, retrosplenial cortex, and parahippocampal cortex were responsive to Euclidean distance from home. These results provide the first evidence of a constantly maintained homing signal in the human brain. In addition, hippocampus, retrosplenial cortex, and parahippocampal cortex, as well as medial prefrontal cortex, were recruited during successful path integration. These findings suggest that dynamic processes recruit hippocampus, retrosplenial cortex, and parahippocampal cortex in support of path integration, including a homing vector system that tracks movement relative to home.

SIGNIFICANCE STATEMENT

Path integration is the continual updating of position and orientation during navigation. Animal studies have identified place cells and grid cells as important for path integration, but underlying models of path integration in humans have rarely been studied. The results of our novel loop closure task are the first to suggest that a homing vector tracks Euclidean distance from the home location, supported by the hippocampus, retrosplenial cortex, and parahippocampal cortex. These findings suggest a potential homing vector mechanism supporting path integration, which recruits hippocampus and retrosplenial cortex to track movement relative to home. These results provide new avenues for computational and animal models by directing attention to homing vector models of path integration, which differ from current movement-tracking models.

Binocular vision: joining up the eyes

To provide a unitary view of the external world, signals from the two eyes must be combined: a new study pinpoints the location in the human brain where the requisite combination occurs.