Differential patterns of 2D location versus depth decoding along the visual hierarchy

Viewpoint adaptation revealed potential representational differences between 2D images and 3D objects

For convenience and experimental control, cognitive science has relied largely on images as stimuli rather than the real, tangible objects encountered in the real world. Recent evidence suggests that the cognitive processing of images may differ from real objects, especially in the processing of spatial locations and actions, thought to be mediated by the dorsal visual stream. Perceptual and semantic processing in the ventral visual stream, however, has been assumed to be largely unaffected by the realism of objects. Several studies have found that one key difference accounting for differences between real objects and images is actability; however, less research has investigated another potential difference - the three-dimensional nature of real objects as conveyed by cues like binocular disparity. To investigate the extent to which perception is affected by the realism of a stimulus, we compared viewpoint adaptation when stimuli (a face or a kettle) were 2D (flat images without binocular disparity) vs. 3D (i.e., real, tangible objects or stereoscopic images with binocular disparity). For both faces and kettles, adaptation to 3D stimuli induced stronger viewpoint aftereffects than adaptation to 2D images when the adapting orientation was rightward. A computational model suggested that the difference in aftereffects could be explained by broader viewpoint tuning for 3D compared to 2D stimuli. Overall, our finding narrowed the gap between understanding the neural processing of visual images and real-world objects by suggesting that compared to 2D images, real and simulated 3D objects evoke more broadly tuned neural representations, which may result in stronger viewpoint invariance.

The dominance of spatial information in object identity judgments: A persistent congruency bias even amidst conflicting statistical regularities

Previous studies have posited that spatial location plays a special role in object recognition. Notably, the "spatial congruency bias (SCB)" is a tendency to report objects as the same identity if they are presented at the same location, compared to different locations. Here we found that even when statistical regularities were manipulated in the opposite direction (objects in the same location were three times more likely to be different identities), subjects still exhibited a robust SCB (more likely to report them as the same identity). We replicated this finding across two preregistered experiments. Only in a third experiment where we explicitly informed subjects of the manipulation did the SCB disappear, though the lack of a significantly reversed bias suggests the ingrained congruency bias was not completely overcome. The inclusion of catch trials where the second object was completely masked further bolsters previous evidence that the congruency bias is perceptual, not simply a guessing strategy. These results reinforce the dominant role of spatial information during object recognition and present the SCB as a strong perceptual phenomenon that is incredibly hard to overcome even in the face of opposing regularities and explicit instruction. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Multidimensional Scaling of Cognitive Ability and Academic Achievement Scores

Multidimensional scaling (MDS) was used as an alternate multivariate procedure for investigating intelligence and academic achievement test score correlations. Correlation coefficients among Wechsler Intelligence Scale for Children, Fifth Edition (WISC-5) and Wechsler Individual Achievement Test, Third Edition (WIAT-III) validity sample scores and among Kaufman Assessment Battery for Children, Second Edition (KABC-II) and Kaufman Test of Educational Achievement, Second Edition (KTEA-2) co-norming sample scores were analyzed using multidimensional scaling (MDS). Three-dimensional MDS configurations were the best fit for interpretation in both datasets. Subtests were more clearly organized by CHC ability and academic domain instead of complexity. Auditory-linguistic, figural-visual, reading-writing, and quantitative-numeric regions were visible in all models. Results were mostly similar across different grade levels. Additional analysis with WISC-V and WIAT-III tests showed that content (verbal, numeric, figural) and response process facets (verbal, manual, paper-pencil) were also useful in explaining test locations. Two implications from this study are that caution may be needed when interpreting fluency scores across academic areas, and MDS provides more empirically based validity evidence regarding content and response mode processes.

The Effects of Normal Aging, Subjective Cognitive Decline, Mild Cognitive Impairment, or Alzheimer’s Disease on Visual Search

Background: Alzheimer’s disease (AD) has been confirmed as an influencing factor of visual impairment, but potential concomitant effects on visual and cognitive performance are not well understood. Objective: To provide a new method for early screening of Alzheimer’s disease and further explore the theoretical mechanism of the decline of whole visual and cognitive performance in AD. Methods: We studied 60 individuals without dementia as normal control (NC), 74 individuals with subjective cognitive decline (SCD), 60 individuals with amnesia mild cognitive impairment (aMCI), and 75 patients with AD on a battery of tests designed to measure multiple aspects of basic and higher-order visual perception and cognition. All subjects performed on same visual and cognitive test batteries. Results: The results showed both of four groups, with the stimulus-presentation time being longer, the visual-search performance improved, and both the eye interest-area first fixation duration and the interest-area-fixation count increased. Particularly under the noise-masking condition, the AD group performed the worst at stimulus-presentation times between 300 and 900 ms. The aMCI group, but not the SCD group, performed worse than the NC group at the stimulus-presentation time of either 300 or 500 ms. The interest-area-fixation count was higher in all the patient groups than that in the NC group, and distinguishable between participants with AD and those with SCD or aMCI. Conclusion: The visual-search performance combined with eye-movement tracking under the noise-masking condition can be used for distinguishing AD from normal aging, SCD, and aMCI.

"Scene" from inside: The representation of Observer's space in high-level visual cortex

Human observers are remarkably adept at perceiving and interacting with visual stimuli around them. Compared to visual stimuli like objects or faces, scenes are unique in that they provide enclosures for observers. An observer looks at a scene by being physically inside the scene. The current research explored this unique observer-scene relationship by studying the neural representation of scenes' spatial boundaries. Previous studies hypothesized that scenes' boundaries were processed in sets of high-level visual cortices. Notably, the parahippocampal place area (PPA), exhibited neural sensitivity to scenes that had closed vs. open spatial boundaries (Kravitz et al., 2011; Park et al., 2011). We asked whether this sensitivity reflected the openness of landscape (e.g., forest vs. beach), or the openness of the environment immediately surrounding the observer (i.e., whether a scene was viewed from inside vs. outside a room). Across two human fMRI experiments, we found that the PPA, as well as another well-known navigation-processing area, the occipital place area (OPA), processed scenes' boundaries according to the observer's space rather than the landscape. Moreover, we found that the PPA's activation pattern was susceptible to manipulations involving mid-level perceptual properties of scenes (e.g., rectilinear pattern of window frames), while the OPA's response was not. Our results have important implications for research in visual scene processing and suggest an important role of an observer's location in representing the spatial boundary, beyond the low-level visual input of a landscape.

The Treachery of Images: How Realism Influences Brain and Behavior

Although the cognitive sciences aim to ultimately understand behavior and brain function in the real world, for historical and practical reasons, the field has relied heavily on artificial stimuli, typically pictures. We review a growing body of evidence that both behavior and brain function differ between image proxies and real, tangible objects. We also propose a new framework for immersive neuroscience to combine two approaches: (i) the traditional build-up approach of gradually combining simplified stimuli, tasks, and processes; and (ii) a newer tear-down approach that begins with reality and compelling simulations such as virtual reality to determine which elements critically affect behavior and brain processing.

A functional theory of bistable perception based on dynamical circular inference

When we face ambiguous images, the brain cannot commit to a single percept; instead, it switches between mutually exclusive interpretations every few seconds, a phenomenon known as bistable perception. While neuromechanistic models, e.g., adapting neural populations with lateral inhibition, may account for the dynamics of bistability, a larger question remains unresolved: how this phenomenon informs us on generic perceptual processes in less artificial contexts. Here, we propose that bistable perception is due to our prior beliefs being reverberated in the cortical hierarchy and corrupting the sensory evidence, a phenomenon known as “circular inference”. Such circularity could occur in a hierarchical brain where sensory responses trigger activity in higher-level areas but are also modulated by feedback projections from these same areas. We show that in the face of ambiguous sensory stimuli, circular inference can change the dynamics of the perceptual system and turn what should be an integrator of inputs into a bistable attractor switching between two highly trusted interpretations. The model captures various aspects of bistability, including Levelt’s laws and the stabilizing effects of intermittent presentation of the stimulus. Since it is related to the generic perceptual inference and belief updating mechanisms, this approach can be used to predict the tendency of individuals to form aberrant beliefs from their bistable perception behavior. Overall, we suggest that feedforward/feedback information loops in hierarchical neural networks, a phenomenon that could lead to psychotic symptoms when overly strong, could also underlie perception in nonclinical populations.

In cases of high ambiguity, our perceptual system cannot commit to a single percept and switches between different interpretations, giving rise to bistable perception. In this paper we outline a computational model of bistability based on the notion of circular inference, i.e. a form of suboptimal hierarchical inference in which priors and / or sensory inputs are reverberated and over-counted. We suggest that descending loops (i.e. reverberated priors) transform our perceptual system from a simple accumulator of sensory inputs into a bistable attractor, that switches between two highly-trusted interpretations. Using analytical methods we derive the necessary conditions for bistable perception to occur. We show that our dynamical circular inference model is able to capture many features of bistability, such as Levelt’s laws and the stabilizing effects of intermittent presentation of the stimulus. Finally we make novel predictions about the behavior of psychotic patients.

3D shape estimation in a constraint optimization neural network

One of the most important aspects of visual perception is the inference of 3D shape from a 2D retinal image of the outside world. The existence of several valid mapping functions from object to data makes this inverse problem ill-posed and therefore computationally difficult. In human vision, the retinal image is a 2D projection of the 3D world. The visual system imposes certain constraints on the family of solutions in order to uniquely and efficiently solve this inverse problem. This work specifically focused on the minimization of standard deviations of 3D angles (MSDA) for 3D perception. Our goal was to use a Deep Convolutional Neural Network based on biological principles derived from visual area V4 to achieve 3D reconstruction using constrained minimization of MSDA. We conducted an experiment using novel shapes with human subjects to collect data and test the model. The performance of the network largely agreed with how humans estimated novel 3D shapes. The results show that the constraint of MSDA in 3D shape can be implemented in a neural network and produce human-like results. Additional visual constraints can be added to the network in the future to fully test the theory of visual constraints as a basis of 3D shape perception.

The influence of spatial location on same-different judgments of facial identity and expression

The "spatial congruency bias" is a behavioral phenomenon where 2 objects presented sequentially are more likely to be judged as being the same object if they are presented in the same location (Golomb, Kupitz, & Thiemann, 2014), suggesting that irrelevant spatial location information may be bound to object representations. Here, we examine whether the spatial congruency bias extends to higher-level object judgments of facial identity and expression. On each trial, 2 real-world faces were sequentially presented in variable screen locations, and subjects were asked to make same-different judgments on the facial expression (Experiments 1-2) or facial identity (Experiment 3) of the stimuli. We observed a robust spatial congruency bias for judgments of facial identity, yet a more fragile one for judgments of facial expression. Subjects were more likely to judge 2 faces as displaying the same expression if they were presented in the same location (compared to in different locations), but only when the faces shared the same identity. On the other hand, a spatial congruency bias was found when subjects made judgments on facial identity, even across faces displaying different facial expressions. These findings suggest a possible difference between the binding of facial identity and facial expression to spatial location. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Relation matters: relative depth order is stored in working memory for depth

Working memory is considered as a cognitive memory buffer for temporarily holding, processing, and manipulating information. Although working memory for verbal and visual information has been studied extensively in the past literature, few studies have systematically investigated how depth information is stored in working memory. Here, we show that the memory performance for detecting changes in stereoscopic depth is low when there is no change in relative depth order, and the performance is reliably better when depth order is changed. Increasing the magnitude of change only improves memory performance when depth order is kept constant. However, if depth order is changed, the performance remains high, even with a small change magnitude. Our findings suggest that relative depth order is a better indicator for working memory performance than absolute metric depth. The memory representation for individual depth is not independent, but inherently relational, revealing a fundamental organizing principle for depth information in the visual system.

Coding of Navigational Distance and Functional Constraint of Boundaries in the Human Scene-Selective Cortex

For visually guided navigation, the use of environmental cues is essential. Particularly, detecting local boundaries that impose limits to locomotion and estimating their location is crucial. In a series of three fMRI experiments, we investigated whether there is a neural coding of navigational distance in the human visual cortex (both female and male). We used virtual reality software to systematically manipulate the distance from a viewer perspective to different types of a boundary. Using a multivoxel pattern classification employing a linear support vector machine, we found that the occipital place area (OPA) is sensitive to the navigational distance restricted by the transparent glass wall. Further, the OPA was sensitive to a non-crossable boundary only, suggesting an importance of the functional constraint of a boundary. Together, we propose the OPA as a perceptual source of external environmental features relevant for navigation.SIGNIFICANCE STATEMENT One of major goals in cognitive neuroscience has been to understand the nature of visual scene representation in human ventral visual cortex. An aspect of scene perception that has been overlooked despite its ecological importance is the analysis of space for navigation. One of critical computation necessary for navigation is coding of distance to environmental boundaries that impose limit on navigator's movements. This paper reports the first empirical evidence for coding of navigational distance in the human visual cortex and its striking sensitivity to functional constraint of environmental boundaries. Such finding links the paper to previous neurological and behavioral works that emphasized the distance to boundaries as a crucial geometric property for reorientation behavior of children and other animal species.

Working memory for stereoscopic depth is limited and imprecise-evidence from a change detection task

Most studies on visual working memory (VWM) and spatial working memory (SWM) have employed visual stimuli presented at the fronto-parallel plane and few have involved depth perception. VWM is often considered as a memory buffer for temporarily holding and manipulating visual information that relates to visual features of an object, and SWM for holding and manipulating spatial information that concerns the spatial location of an object. Although previous research has investigated the effect of stereoscopic depth on VWM, the question of how depth positions are stored in working memory has not been systematically investigated, leaving gaps in the existing literature on working memory. Here, we explore working memory for depth by using a change detection task. The memory items were presented at various stereoscopic depth planes perpendicular to the line of sight, with one item per depth plane. Participants were asked to make judgments on whether the depth position of the target (one of the memory items) had changed. The results showed a conservative response bias that observers tended to make 'no change' responses when detecting changes in depth. In addition, we found that similar to VWM, the change detection accuracy degraded with the number of memory items presented, but the accuracy was much lower than that reported for VWM, suggesting that the storage for depth information is severely limited and less precise than that for visual information. The detection sensitivity was higher for the nearest and farthest depths and was better when the probe was presented along with the other items originally in the memory array, indicating that how well the to-be-stored depth can be stored in working memory depends on its relation with the other depth positions.

Decoding disparity categories in 3-dimensional images from fMRI data using functional connectivity patterns

Humans use binocular disparity to extract depth information from two-dimensional retinal images in a process called stereopsis. Previous studies usually introduce the standard univariate analysis to describe the correlation between disparity level and brain activity within a given brain region based on functional magnetic resonance imaging (fMRI) data. Recently, multivariate pattern analysis has been developed to extract activity patterns across multiple voxels for deciphering categories of binocular disparity. However, the functional connectivity (FC) of patterns based on regions of interest or voxels and their mapping onto disparity category perception remain unknown. The present study extracted functional connectivity patterns for three disparity conditions (crossed disparity, uncrossed disparity, and zero disparity) at distinct spatial scales to decode the binocular disparity. Results of 27 subjects' fMRI data demonstrate that FC features are more discriminatory than traditional voxel activity features in binocular disparity classification. The average binary classification of the whole brain and visual areas are respectively 87% and 79% at single subject level, and thus above the chance level (50%). Our research highlights the importance of exploring functional connectivity patterns to achieve a novel understanding of 3D image processing.

Multivariate Analysis of BOLD Activation Patterns Recovers Graded Depth Representations in Human Visual and Parietal Cortex

Navigating through natural environments requires localizing objects along three distinct spatial axes. Information about position along the horizontal and vertical axes is available from an object’s position on the retina, while position along the depth axis must be inferred based on second-order cues such as the disparity between the images cast on the two retinae. Past work has revealed that object position in two-dimensional (2D) retinotopic space is robustly represented in visual cortex and can be robustly predicted using a multivariate encoding model, in which an explicit axis is modeled for each spatial dimension. However, no study to date has used an encoding model to estimate a representation of stimulus position in depth. Here, we recorded BOLD fMRI while human subjects viewed a stereoscopic random-dot sphere at various positions along the depth (z) and the horizontal (x) axes, and the stimuli were presented across a wider range of disparities (out to ∼40 arcmin) compared to previous neuroimaging studies. In addition to performing decoding analyses for comparison to previous work, we built encoding models for depth position and for horizontal position, allowing us to directly compare encoding between these dimensions. Our results validate this method of recovering depth representations from retinotopic cortex. Furthermore, we find convergent evidence that depth is encoded most strongly in dorsal area V3A.

Category-selective areas in human visual cortex exhibit preferences for stimulus depth

Multiple regions in the human brain are dedicated to accomplish the feat of object recognition; yet our brains must also compute the 2D and 3D locations of the objects we encounter in order to make sense of our visual environments. A number of studies have explored how various object category-selective regions are sensitive to and have preferences for specific 2D spatial locations in addition to processing their preferred-stimulus categories, but there is no survey of how these regions respond to depth information. In a blocked functional MRI experiment, subjects viewed a series of category-specific (i.e., faces, objects, scenes) and unspecific (e.g., random moving dots) stimuli with red/green anaglyph glasses. Critically, these stimuli were presented at different depth planes such that they appeared in front of, behind, or at the same (i.e., middle) depth plane as the fixation point (Experiment 1) or simultaneously in front of and behind fixation (i.e., mixed depth; Experiment 2). Comparisons of mean response magnitudes between back, middle, and front depth planes reveal that face and object regions OFA and LOC exhibit a preference for front depths, and motion area MT+ exhibits a strong linear preference for front, followed by middle, followed by back depth planes. In contrast, scene-selective regions PPA and OPA prefer front and/or back depth planes (relative to middle). Moreover, the occipital place area demonstrates a strong preference for "mixed" depth above and beyond back alone, raising potential implications about its particular role in scene perception. Crucially, the observed depth preferences in nearly all areas were evoked irrespective of the semantic stimulus category being viewed. These results reveal that the object category-selective regions may play a role in processing or incorporating depth information that is orthogonal to their primary processing of object category information.

Separating memoranda in depth increases visual working memory performance

Visual working memory is the mechanism supporting the continued maintenance of information after sensory inputs are removed. Although the capacity of visual working memory is limited, memoranda that are spaced farther apart on a 2-D display are easier to remember, potentially because neural representations are more distinct within retinotopically organized areas of visual cortex during memory encoding, maintenance, or retrieval. The impact on memory of spatial separability in depth is less clear, even though depth information is essential to guiding interactions with objects in the environment. On one account, separating memoranda in depth may facilitate performance if interference between items is reduced. However, depth information must be inferred indirectly from the 2-D retinal image, and less is known about how visual cortex represents depth. Thus, an alternative possibility is that separation in depth does not attenuate between-items interference; it may even impair performance, as attention must be distributed across a larger volume of 3-D space. We tested these alternatives using a stereo display while participants remembered the colors of stimuli presented either near or far in the 2-D plane or in depth. Increasing separation in-plane and in depth both enhanced performance. Furthermore, participants who were better able to utilize stereo depth cues showed larger benefits when memoranda were separated in depth, particularly for large memory arrays. The observation that spatial separation in the inferred 3-D structure of the environment improves memory performance, as is the case in 2-D environments, suggests that separating memoranda in depth might reduce neural competition by utilizing cortically separable resources.

2D location biases depth-from-disparity judgments but not vice versa

Visual cognition in our 3D world requires understanding how we accurately localize objects in 2D and depth, and what influence both types of location information have on visual processing. Spatial location is known to play a special role in visual processing, but most of these findings have focused on the special role of 2D location. One such phenomena is the spatial congruency bias (Golomb, Kupitz, & Thiemann, 2014), where 2D location biases judgments of object features but features do not bias location judgments. This paradigm has recently been used to compare different types of location information in terms of how much they bias different types of features. Here we used this paradigm to ask a related question: whether 2D and depth-from-disparity location bias localization judgments for each other. We found that presenting two objects in the same 2D location biased position-in-depth judgments, but presenting two objects at the same depth (disparity) did not bias 2D location judgments. We conclude that an object's 2D location may be automatically incorporated into perception of its depth location, but not vice versa, which is consistent with a fundamentally special role for 2D location in visual processing.