Fundamental failures of shape constancy resulting from cortical anisotropy

Anisotropy of object non-rigidity: High-level perceptual consequences of cortical anisotropy

This study presents instances where variations in a complex, higher-level perceptual phenomenon- an anisotropy in object non-rigidity is explained by the distribution of low-level neural properties in the primary visual cortex. Specifically, we examine the visual interpretation of two rigidly connected rotating circular rings. At speeds where observers predominantly perceive rigid rotation of the rings rotating horizontally, observers perceive only non-rigid wobbling when the image is rotated by 90°. Additionally, vertically rotating rings appear narrower and longer compared to their physically identical horizontally rotating counterparts. We show that these perceived shape changes can be decoded from V1 outputs by considering anisotropies in orientation-selective cells. We then empirically demonstrate that even when the vertically rotating ellipses are widened or the horizontally rotating ellipses are elongated so that the shapes match, the perceived difference in non-rigidity is reduced only by a small amount and increased non-rigidity persists in vertical rotations, suggesting that motion mechanisms also play a role. By incorporating cortical anisotropies into optic flow computations, we show that motion gradients for vertical rotations align more with physical non-rigidity, while horizontal rotations align closer to rigidity, indicating that cortical anisotropies contribute to the heightened perception of non-rigidity when orientation shifts from horizontal to vertical. The study underscores the importance of low-level anisotropies in shaping high-level percepts and raises questions about their evolutionary significance, particularly for shape constancy and motion perception.

Complexity of mental geometry for 3D pose perception

Biological visual systems rely on pose estimation of 3D objects to navigate and interact with their environment, but the neural mechanisms and computations for inferring 3D poses from 2D retinal images are only partially understood, especially where stereo information is missing. We previously presented evidence that humans infer the poses of 3D objects lying centered on the ground by using the geometrical back-transform from retinal images to viewer-centered world coordinates. This model explained the almost veridical estimation of poses in real scenes and the illusory rotation of poses in obliquely viewed pictures, which includes the "pointing out of the picture" phenomenon. Here we test this model for more varied configurations and find that it needs to be augmented. Five observers estimated poses of sloped, elevated, or off-center 3D sticks in each of 16 different poses displayed on a monitor in frontal and oblique views. Pose estimates in scenes and pictures showed remarkable accuracy and agreement between observers, but with a systematic fronto-parallel bias for oblique poses similar to the ground condition. The retinal projection of the pose of an object sloped wrt the ground depends on the slope. We show that observers' estimates can be explained by the back-transform derived for close to the correct slope. The back-transform explanation also applies to obliquely viewed pictures and to off-center objects and elevated objects, making it more likely that observers use internalized perspective geometry to make 3D pose inferences while actively incorporating inferences about other aspects of object placement.

Cone-Opponent Ganglion Cells in the Primate Fovea Tuned to Noncardinal Color Directions

A long-standing question in vision science is how the three cone photoreceptor types-long (L), medium (M), and short (S) wavelength sensitive-combine to generate our perception of color. Hue perception can be described along two opponent axes: red-green and blue-yellow. Psychophysical measurements of color appearance indicate that the cone inputs to the red-green and blue-yellow opponent axes are M vs. L + S and L vs. M + S, respectively. However, the "cardinal directions of color space" revealed by psychophysical measurements of color detection thresholds following adaptation are L vs. M and S vs. L + M. These cardinal directions match the most common cone-opponent retinal ganglion cells (RGCs) in the primate retina. Accordingly, the cone opponency necessary for color appearance is thought to be established in the cortex. While neurons with the appropriate M vs. L + S and L vs. M + S opponency have been reported in the retina and lateral geniculate nucleus, their existence continues to be debated. Resolving this long-standing debate is necessary because a complete account of the cone opponency in the retinal output is critical for understanding how downstream neural circuits process color. Here, we performed adaptive optics calcium imaging to noninvasively measure foveal RGC light responses in the living Macaca fascicularis eye. We confirm the presence of L vs. M + S and M vs. L + S neurons with noncardinal cone opponency and demonstrate that cone-opponent signals in the retinal output are more diverse than classically thought.

Mental geometry of perceiving 3D size in pictures

We show that the classical problem of three-dimensional (3D) size perception in obliquely viewed pictures can be understood by comparing human performance to the optimal geometric solution. A photograph seen from the camera position, can form the same retinal projection as the physical 3D scene, but retinal projections of sizes and shapes are distorted in oblique viewing. For real scenes, we previously showed that size and shape inconstancy result despite observers using the correct geometric back-transform, because some retinal images evoke misestimates of object slant or viewing elevation. Now, we examine how observers estimate 3D sizes in oblique views of pictures of objects lying on the ground in different poses. Compared to estimates for real scenes, in oblique views of pictures, sizes were seriously underestimated for objects at frontoparallel poses, but there was almost no change for objects perceived as pointing toward the viewer. The inverse of the function relating projected length to pose, camera elevation and viewing azimuth, gives the optimal correction factor for inferring correct 3D lengths if the elevation and azimuth are estimated accurately. Empirical correction functions had similar shapes to optimal, but lower amplitude. Measurements revealed that observers systematically underestimated viewing azimuth, similar to the frontoparallel bias for object pose perception. A model that adds underestimation of viewing azimuth to the geometrical back-transform, provided good fits to estimated 3D lengths from oblique views. These results add to accumulating evidence that observers use internalized projective geometry to perceive sizes, shapes, and poses in 3D scenes and their pictures.

Mental geometry of three-dimensional size perception

Judging the poses, sizes, and shapes of objects accurately is necessary for organisms and machines to operate successfully in the world. Retinal images of three-dimensional objects are mapped by the rules of projective geometry and preserve the invariants of that geometry. Since Plato, it has been debated whether geometry is innate to the human brain, and Poincare and Einstein thought it worth examining whether formal geometry arises from experience with the world. We examine if humans have learned to exploit projective geometry to estimate sizes and aspects of three-dimensional shape that are related to relative lengths and aspect ratios.

Numerous studies have examined size invariance as a function of physical distance, which changes scale on the retina. However, it is surprising that possible constancy or inconstancy of relative size seems not to have been investigated for object pose, which changes retinal image size differently along different axes. We show systematic underestimation of length for extents pointing toward or away from the observer, both for static objects and dynamically rotating objects. Observers do correct for projected shortening according to the optimal back-transform, obtained by inverting the projection function, but the correction is inadequate by a multiplicative factor. The clue is provided by the greater underestimation for longer objects, and the observation that they seem to be more slanted toward the observer. Adding a multiplicative factor for perceived slant in the back-transform model provides good fits to the corrections used by observers. We quantify the slant illusion with two different slant matching measurements, and use a dynamic demonstration to show that the slant illusion perceptually dominates length nonrigidity.

In biological and mechanical objects, distortions of shape are manifold, and changes in aspect ratio and relative limb sizes are functionally important. Our model shows that observers try to retain invariance of these aspects of shape to three-dimensional rotation by correcting retinal image distortions due to perspective projection, but the corrections can fall short. We discuss how these results imply that humans have internalized particular aspects of projective geometry through evolution or learning, and if humans assume that images are preserving the continuity, collinearity, and convergence invariances of projective geometry, that would simply explain why illusions such as Ames’ chair appear cohesive despite being a projection of disjointed elements, and thus supplement the generic viewpoint assumption.

Picture perception reveals mental geometry of 3D scene inferences

Pose estimation of objects in real scenes is critically important for biological and machine visual systems, but little is known of how humans infer 3D poses from 2D retinal images. We show unexpectedly remarkable agreement in the 3D poses different observers estimate from pictures. We further show that all observers apply the same inferential rule from all viewpoints, utilizing the geometrically derived back-transform from retinal images to actual 3D scenes. Pose estimations are altered by a fronto-parallel bias, and by image distortions that appear to tilt the ground plane. We used pictures of single sticks or pairs of joined sticks taken from different camera angles. Observers viewed these from five directions, and matched the perceived pose of each stick by rotating an arrow on a horizontal touchscreen. The projection of each 3D stick to the 2D picture, and then onto the retina, is described by an invertible trigonometric expression. The inverted expression yields the back-projection for each object pose, camera elevation, and observer viewpoint. We show that a model that uses the back-projection, modulated by just two free parameters, explains 560 pose estimates per observer. By considering changes in retinal image orientations due to position and elevation of limbs, the model also explains perceived limb poses in a complex scene of two bodies lying on the ground. The inferential rules simply explain both perceptual invariance and dramatic distortions in poses of real and pictured objects, and show the benefits of incorporating projective geometry of light into mental inferences about 3D scenes.

Functional implications of orientation maps in primary visual cortex

Stimulus orientation in the primary visual cortex of primates and carnivores is mapped as iso-orientation domains radiating from pinwheel centres, where orientation preferences of neighbouring cells change circularly. Whether this orientation map has a function is currently debated, because many mammals, such as rodents, do not have such maps. Here we show that two fundamental properties of visual cortical responses, contrast saturation and cross-orientation suppression, are stronger within cat iso-orientation domains than at pinwheel centres. These differences develop when excitation (not normalization) from neighbouring oriented neurons is applied to different cortical orientation domains and then balanced by inhibition from un-oriented neurons. The functions of the pinwheel mosaic emerge from these local intra-cortical computations: Narrower tuning, greater cross-orientation suppression and higher contrast gain of iso-orientation cells facilitate extraction of object contours from images, whereas broader tuning, greater linearity and less suppression of pinwheel cells generate selectivity for surface patterns and textures.

Stimulus orientation in the primary visual cortex of primates and carnivores is mapped into a geometrical mosaic but the functional implications of these maps remain debated. Here the authors reveal an association between the structure of cortical orientation maps in cats, and the functions of local cortical circuits in processing patterns and contours.

Cardinal rules: visual orientation perception reflects knowledge of environmental statistics

Humans are good at performing visual tasks, but experimental measurements have revealed substantial biases in the perception of basic visual attributes. An appealing hypothesis is that these biases arise through a process of statistical inference, in which information from noisy measurements is fused with a probabilistic model of the environment. However, such inference is optimal only if the observer's internal model matches the environment. We found this to be the case. We measured performance in an orientation-estimation task and found that orientation judgments were more accurate at cardinal (horizontal and vertical) orientations. Judgments made under conditions of uncertainty were strongly biased toward cardinal orientations. We estimated observers' internal models for orientation and found that they matched the local orientation distribution measured in photographs. In addition, we determined how a neural population could embed probabilistic information responsible for such biases.

The utility of shape attributes in deciphering movements of non-rigid objects

Most moving objects in the world are non-rigid, changing shape as they move. To disentangle shape changes from movements, computational models either fit shapes to combinations of basis shapes or motion trajectories to combinations of oscillations but are biologically unfeasible in their input requirements. Recent neural models parse shapes into stored examples, which are unlikely to exist for general shapes. We propose that extracting shape attributes, e.g., symmetry, facilitates veridical perception of non-rigid motion. In a new method, identical dots were moved in and out along invisible spokes, to simulate the rotation of dynamically and randomly distorting shapes. Discrimination of rotation direction measured as a function of non-rigidity was 90% as efficient as the optimal Bayesian rotation decoder and ruled out models based on combining the strongest local motions. Remarkably, for non-rigid symmetric shapes, observers outperformed the Bayesian model when perceived rotation could correspond only to rotation of global symmetry, i.e., when tracking of shape contours or local features was uninformative. That extracted symmetry can drive perceived motion suggests that shape attributes may provide links across the dorsal-ventral separation between motion and shape processing. Consequently, the perception of non-rigid object motion could be based on representations that highlight global shape attributes.