Size contrast and assimilation explained by the statistics of natural scene geometry

Effect of spatial context on perceived walking direction

Contextual modulation occurs for many aspects of high-level vision but is relatively unexplored for the perception of walking direction. In a recent study, we observed an effect of the temporal context on perceived walking direction. Here, we examined the spatial contextual modulation by measuring the perceived direction of a target point-light walker in the presence of two flanker walkers, one on each side. Experiment 1 followed a within-subjects design. Participants (n = 30) completed a spatial context task by judging the walking direction of the target in 13 different conditions: a walker alone in the center or with two flanking walkers either intact or scrambled at a flanker deviation of ±15°, ±30°, or ±45°. For comparison, participants completed an adaptation task where they reported the walking direction of a target after adaptation to ±30° walking direction. We found the expected repulsive effects in the adaptation task but attractive effects in the spatial context task. In Experiment 2 (n = 40), we measured the tuning of spatial contextual modulation across a wide range of flanker deviation magnitudes ranging from 15° to 165° in 15° intervals. Our results showed significant attractive effects across a wide range of flanker walking directions with the peak effect at around 30°. The assimilative versus repulsive effects of spatial contextual modulation and temporal adaptation suggest dissociable neural mechanisms, but they may operate on the same population of sensory channels coding for walking direction, as evidenced by similarity in the peak tuning across the walking direction of the inducers.

View Normalization of Object Size in the Right Parietal Cortex

Prior knowledge alters perception already on early levels of processing. For instance, judging the display size of an object is affected by its familiar size. Using functional magnetic resonance imaging, we investigated the neural processes involved in resolving ambiguities between familiar object size and physical object size in 33 healthy human subjects. The familiar size was either small or large, and the object was displayed as either small or large. Thus, the size of the displayed object was either congruent or incongruent with its internally stored canonical size representation. Subjects were asked to indicate where the stimuli appeared on the screen as quickly and accurately as possible, thereby ensuring that differential activations cannot be ascribed to explicit object size judgments. Incongruent (relative to congruent) object displays were associated with enhanced activation of the right intraparietal sulcus (IPS). These data are consistent with but extend previous patient studies, which found the right parietal cortex involved in matching visual objects presented atypically to prototypical object representations, suggesting that the right IPS supports view normalization of objects. In a second experiment, using a parametric design, a region-of-interest analysis supported this notion by showing that increases in size mismatch between the displayed size of an object and its familiar viewing size were associated with an increased right IPS activation. We conclude that the right IPS performs view normalization of mismatched information about the internally stored prototypical size and the current viewing size of an object.

Individual differences in the Müller-Lyer and Ponzo illusions are stable across different contexts

Vision scientists have attempted to classify visual illusions according to certain aspects, such as brightness or spatial features. For example, Piaget proposed that visual illusion magnitudes either decrease or increase with age. Subsequently, it was suggested that illusions are segregated according to their context: real-world contexts enhance and abstract contexts inhibit illusion magnitudes with age. We tested the effects of context on the Müller-Lyer and Ponzo illusions with a standard condition (no additional context), a line-drawing perspective condition, and a real-world perspective condition. A mixed-effects model analysis, based on data from 76 observers with ages ranging from 6 to 66 years, did not reveal any significant interaction between context and age. Although we found strong intra-illusion correlations for both illusions, we found only weak inter-illusion correlations, suggesting that the structure underlying these two spatial illusions includes several specific factors.

A Perceptual Inference Mechanism for Hallucinations Linked to Striatal Dopamine

Hallucinations, a cardinal feature of psychotic disorders such as schizophrenia, are known to depend on excessive striatal dopamine. However, an underlying cognitive mechanism linking dopamine dysregulation and the experience of hallucinatory percepts remains elusive. Bayesian models explain perception as an optimal combination of prior expectations and new sensory evidence, where perceptual distortions such as illusions and hallucinations may occur if prior expectations are afforded excessive weight. Such excessive weight of prior expectations, in turn, could stem from a gain-control process controlled by neuromodulators such as dopamine. To test for such a dopamine-dependent gain-control mechanism of hallucinations, we studied unmedicated patients with schizophrenia with varying degrees of hallucination severity and healthy individuals using molecular imaging with a pharmacological manipulation of dopamine, structural imaging, and a novel task designed to measure illusory changes in the perceived duration of auditory stimuli under different levels of uncertainty. Hallucinations correlated with a perceptual bias, reflecting disproportional gain on expectations under uncertainty. This bias could be pharmacologically induced by amphetamine, strongly correlated with striatal dopamine release, and related to cortical volume of the dorsal anterior cingulate, a brain region involved in tracking environmental uncertainty. These findings outline a novel dopamine-dependent mechanism for perceptual modulation in physiological conditions and further suggest that this mechanism may confer vulnerability to hallucinations in hyper-dopaminergic states underlying psychosis.

The snow tire illusion: Different levels of perceptual assimilation across a single stimulus configuration

While observing a specific traffic sign in the field, we noticed an apparent distortion of size and shape of the circle that contained the sign. This novel illusion manifests as a distortion of the horizontal compared to the vertical dimension of the sign. The illusion seems to be underlined by similar mechanisms to those in the Delboeuf illusion. The aim of our study was to determine the existence and magnitude of the snow tire illusion. We conducted two experiments using the method of constant stimuli. The first experiment was conducted on the standard sign, while in the second, the stimuli were rotated 90° counterclockwise. Both experiments consisted of three conditions: the snow tire, the ellipse, and the simple circle (control) conditions. The data showed a robust illusion effect for both the standard and rotated sign compared to the control condition, with a large majority of participants experiencing the illusion. The snow tire illusion seems to be a combination of assimilation mechanisms of different magnitudes. The assimilation is larger for one dimension of the sign, thus producing the shape distortion. The illusion may be a manifestation of a thus far undocumented non-uniform effect of assimilation on perceived size and shape.

Statistics of high-level scene context

CONTEXT IS CRITICAL FOR RECOGNIZING ENVIRONMENTS AND FOR SEARCHING FOR OBJECTS WITHIN THEM: contextual associations have been shown to modulate reaction time and object recognition accuracy, as well as influence the distribution of eye movements and patterns of brain activations. However, we have not yet systematically quantified the relationships between objects and their scene environments. Here I seek to fill this gap by providing descriptive statistics of object-scene relationships. A total of 48, 167 objects were hand-labeled in 3499 scenes using the LabelMe tool (Russell et al., 2008). From these data, I computed a variety of descriptive statistics at three different levels of analysis: the ensemble statistics that describe the density and spatial distribution of unnamed "things" in the scene; the bag of words level where scenes are described by the list of objects contained within them; and the structural level where the spatial distribution and relationships between the objects are measured. The utility of each level of description for scene categorization was assessed through the use of linear classifiers, and the plausibility of each level for modeling human scene categorization is discussed. Of the three levels, ensemble statistics were found to be the most informative (per feature), and also best explained human patterns of categorization errors. Although a bag of words classifier had similar performance to human observers, it had a markedly different pattern of errors. However, certain objects are more useful than others, and ceiling classification performance could be achieved using only the 64 most informative objects. As object location tends not to vary as a function of category, structural information provided little additional information. Additionally, these data provide valuable information on natural scene redundancy that can be exploited for machine vision, and can help the visual cognition community to design experiments guided by statistics rather than intuition.

Cognitive architecture of perceptual organization: from neurons to gnosons

What, if anything, is cognitive architecture and how is it implemented in neural architecture? Focusing on perceptual organization, this question is addressed by way of a pluralist approach which, supported by metatheoretical considerations, combines complementary insights from representational, connectionist, and dynamic systems approaches to cognition. This pluralist approach starts from a representationally inspired model which implements the intertwined but functionally distinguishable subprocesses of feedforward feature encoding, horizontal feature binding, and recurrent feature selection. As sustained by a review of neuroscientific evidence, these are the subprocesses that are believed to take place in the visual hierarchy in the brain. Furthermore, the model employs a special form of processing, called transparallel processing, whose neural signature is proposed to be gamma-band synchronization in transient horizontal neural assemblies. In neuroscience, such assemblies are believed to mediate binding of similar features. Their formal counterparts in the model are special input-dependent distributed representations, called hyperstrings, which allow many similar features to be processed in a transparallel fashion, that is, simultaneously as if only one feature were concerned. This form of processing does justice to both the high combinatorial capacity and the high speed of the perceptual organization process. A naturally following proposal is that those temporarily synchronized neural assemblies are "gnosons", that is, constituents of flexible self-organizing cognitive architecture in between the relatively rigid level of neurons and the still elusive level of consciousness.

An empirical explanation of aperture effects

The perceived direction of a moving line changes, often markedly, when viewed through an aperture. Although several explanations of this remarkable effect have been proposed, these accounts typically focus on the percepts elicited by a particular type of aperture and offer no biological rationale. Here, we test the hypothesis that to contend with the inherently ambiguous nature of motion stimuli the perceived direction of objects moving behind apertures of different shapes is determined by a wholly empirical strategy of visual processing. An analysis of moving line stimuli generated by objects projected through apertures shows that the directions of motion subjects report in psychophysical testing is accounted for by the frequency of occurrence of the 2D directions of stimuli generated by simulated 3D sources. The completeness of these predictions supports the conclusion that the direction of perceived motion is fully determined by accumulated behavioral experience with sources whose physical motions cannot be conveyed by image sequences as such.

The role of hand size in the fake-hand illusion paradigm

When a hand (either real or fake) is stimulated in synchrony with our own hand concealed from view, the felt position of our own hand can be biased toward the location of the seen hand. This intriguing phenomenon relies on the brain's ability to detect statistical correlations in the multisensory inputs (ie visual, tactile, and proprioceptive), but it is also modulated by the pre-existing representation of one's own body. Nonetheless, researchers appear to have accepted the assumption that the size of the seen hand does not matter for this illusion to occur. Here we used a real-time video image of the participant's own hand to elicit the illusion, but we varied the hand size in the video image so that the seen hand was either reduced, veridical, or enlarged in comparison to the participant's own hand. The results showed that visible-hand size modulated the illusion, which was present for veridical and enlarged images of the hand, but absent when the visible hand was reduced. These findings indicate that very specific aspects of our own body image (ie hand size) can constrain the multisensory modulation of the body schema highlighted by the fake-hand illusion paradigm. In addition, they suggest an asymmetric tendency to acknowledge enlarged (but not reduced) images of body parts within our body representation.

What are lightness illusions and why do we see them?

Lightness illusions are fundamental to human perception, and yet why we see them is still the focus of much research. Here we address the question by modelling not human physiology or perception directly as is typically the case but our natural visual world and the need for robust behaviour. Artificial neural networks were trained to predict the reflectance of surfaces in a synthetic ecology consisting of 3-D “dead-leaves” scenes under non-uniform illumination. The networks learned to solve this task accurately and robustly given only ambiguous sense data. In addition—and as a direct consequence of their experience—the networks also made systematic “errors” in their behaviour commensurate with human illusions, which includes brightness contrast and assimilation—although assimilation (specifically White's illusion) only emerged when the virtual ecology included 3-D, as opposed to 2-D scenes. Subtle variations in these illusions, also found in human perception, were observed, such as the asymmetry of brightness contrast. These data suggest that “illusions” arise in humans because (i) natural stimuli are ambiguous, and (ii) this ambiguity is resolved empirically by encoding the statistical relationship between images and scenes in past visual experience. Since resolving stimulus ambiguity is a challenge faced by all visual systems, a corollary of these findings is that human illusions must be experienced by all visual animals regardless of their particular neural machinery. The data also provide a more formal definition of illusion: the condition in which the true source of a stimulus differs from what is its most likely (and thus perceived) source. As such, illusions are not fundamentally different from non-illusory percepts, all being direct manifestations of the statistical relationship between images and scenes.

Sometimes the best way to understand how the visual brain works is to understand why it sometimes does not. Thus, visual illusions have been central to the science and philosophy of human consciousness for decades. Here we explain the root cause of brightness illusions, not by modelling human perception or its assumed physiological substrate (as is more typically done), but by modelling the basic challenge that all visual animals must resolve if they are to survive: the inherent ambiguity of sensory data. We do this by training synthetic neural networks to recognise surfaces under different lights in scenes with naturalistic structure. The result is that the networks not only solve this task robustly (i.e., they exhibit “lightness constancy”), they also—as a consequence—exhibit the same illusions of lightness that humans also see. In short, these synthetic systems not only get it right like we do, but also get it wrong like we do, too. This emergent coincidence strongly provides causal evidence that illusions (and by extension all percepts) represent the probable source of images in past visual experience, which has fundamental consequences for explaining how and why we see what we do. The study also suggests the first formal definition of what an illusion is: The condition in which the actual source of a stimulus differs from its most likely source.

First-order and second-order statistical analysis of 3D and 2D image structure

In the first part of this article, we analyze the relation between local image structures (i.e., homogeneous, edge-like, corner-like or texture-like structures) and the underlying local 3D structure (represented in terms of continuous surfaces and different kinds of 3D discontinuities) using range data with real-world color images. We find that homogeneous image structures correspond to continuous surfaces, and discontinuities are mainly formed by edge-like or corner-like structures, which we discuss regarding potential computer vision applications and existing assumptions about the 3D world. In the second part, we utilize the measurements developed in the first part to investigate how the depth at homogeneous image structures is related to the depth of neighbor edges. For this, we first extract the local 3D structure of regularly sampled points, and then, analyze the coplanarity relation between these local 3D structures. We show that the likelihood to find a certain depth at a homogeneous image patch depends on the distance between the image patch and a neighbor edge. We find that this dependence is higher when there is a second neighbor edge which is coplanar with the first neighbor edge. These results allow deriving statistically based prediction models for depth interpolation on homogeneous image structures.

Evolution of visually guided behavior in artificial agents

Recent work on brightness, color, and form has suggested that human visual percepts represent the probable sources of retinal images rather than stimulus features as such. Here we investigate the plausibility of this empirical concept of vision by allowing autonomous agents to evolve in virtual environments based solely on the relative success of their behavior. The responses of evolved agents to visual stimuli indicate that fitness improves as the neural network control systems gradually incorporate the statistical relationship between projected images and behavior appropriate to the sources of the inherently ambiguous images. These results: (1) demonstrate the merits of a wholly empirical strategy of animal vision as a means of contending with the inverse optics problem; (2) argue that the information incorporated into biological visual processing circuitry is the relationship between images and their probable sources; and (3) suggest why human percepts do not map neatly onto physical reality.

Comparison of Bayesian and empirical ranking approaches to visual perception

Much current vision research is predicated on the idea--and a rapidly growing body of evidence--that visual percepts are generated according to the empirical significance of light stimuli rather than their physical characteristics. As a result, an increasing number of investigators have asked how visual perception can be rationalized in these terms. Here, we compare two different theoretical frameworks for predicting what observers actually see in response to visual stimuli: Bayesian decision theory and empirical ranking theory. Deciding which of these approaches has greater merit is likely to determine how the statistical operations that apparently underlie visual perception are eventually understood.

The Poggendorff illusion explained by natural scene geometry

One of the most intriguing of the many discrepancies between perceived spatial relationships and the physical structure of visual stimuli is the Poggendorff illusion, when an obliquely oriented line that is interrupted no longer appears collinear. Although many different theories have been proposed to explain this effect, there has been no consensus about its cause. Here, we use a database of range images (i.e., images that include the distance from the image plane of every pixel in the scene) to show that the probability distribution of the possible locations of line segments across an interval in natural environments can fully account for all of the behavior of this otherwise puzzling phenomenon.

The Müller-Lyer illusion explained by the statistics of image-source relationships

The Müller-Lyer effect, the apparent difference in the length of a line as the result of its adornment with arrowheads or arrow tails, is the best known and most controversial of the classical geometrical illusions. By sampling a range-image database of natural scenes, we show that the perceptual effects elicited by the Müller-Lyer stimulus and its major variants are correctly predicted by the probability distributions of the possible physical sources underlying the relevant retinal images. These results support the conclusion that the Müller-Lyer illusion is a manifestation of the probabilistic strategy of visual processing that has evolved to contend with the uncertain provenance of retinal stimuli.

Natural-scene geometry predicts the perception of angles and line orientation

Visual stimuli that entail the intersection of two or more straight lines elicit a variety of well known perceptual anomalies. Preeminent among these anomalies are the systematic overestimation of acute angles, the underestimation of obtuse angles, and the misperceptions of line orientation exemplified in the classical tilt, Zollner, and Hering illusions. Here we show that the probability distributions of the possible real-world sources of projected lines and angles derived from a range-image database of natural scenes accurately predict each of these perceptual peculiarities. These findings imply that the perception of angles and oriented lines is determined by the statistical relationship between geometrical stimuli and their physical sources in typical visual environments.