'Proto-rivalry': how the binocular brain identifies gloss

Learning to see stuff

Materials with complex appearances, like textiles and foodstuffs, pose challenges for conventional theories of vision. But recent advances in unsupervised deep learning provide a framework for explaining how we learn to see them. We suggest that perception does not involve estimating physical quantities like reflectance or lighting. Instead, representations emerge from learning to encode and predict the visual input as efficiently and accurately as possible. Neural networks can be trained to compress natural images or to predict frames in movies without 'ground truth' data about the outside world. Yet, to succeed, such systems may automatically discover how to disentangle distal causal factors. Such 'statistical appearance models' potentially provide a coherent explanation of both failures and successes in perception.

Gloss and Speed Judgments Yield Different Fine Tuning of Saccadic Sampling in Dynamic Scenes

Image motion contains potential cues about the material properties of objects. In earlier work, we proposed motion cues that could predict whether a moving object would be perceived as shiny or matte. However, whether the visual system uses these cues is still uncertain. Herein, we use the tracking of eye movements as a tool to understand what visual information observers use when engaged in material perception. Observers judged either the gloss or the speed of moving blobby shapes in an eye tracking experiment. Results indicate that during glossiness judgments, participants tend to look at gloss-diagnostic dynamic features more than during speed judgments. This suggests a fine tuning of the visual system to properties of moving stimuli: Task relevant information is actively singled out and processed in a dynamically changing environment.

Reflections on glass

An important phenomenon in the study of human perception is the ability of observers to identify different types of surface materials. The present article will consider a wide range of factors that can influence the perceptual identification of glass, including the structural complexity of an object, whether it is hollow or solid, and the pattern of illumination. Several illumination techniques used in the field of photography are described, and examples are provided to show how they interact with structural complexity. A single psychophysical experiment is reported to evaluate the perceptions of naïve observers using a novel categorization task designed to assess potential confusions among multiple material categories. Finally, the paper will enumerate a number of specific image features that are potentially diagnostic for the identification of glass, and it will evaluate their relative importance for human perception.

Material Perception

Under typical viewing conditions, human observers effortlessly recognize materials and infer their physical, functional, and multisensory properties at a glance. Without touching materials, we can usually tell whether they would feel hard or soft, rough or smooth, wet or dry. We have vivid visual intuitions about how deformable materials like liquids or textiles respond to external forces and how surfaces like chrome, wax, or leather change appearance when formed into different shapes or viewed under different lighting. These achievements are impressive because the retinal image results from complex optical interactions between lighting, shape, and material, which cannot easily be disentangled. Here I argue that because of the diversity, mutability, and complexity of materials, they pose enormous challenges to vision science: What is material appearance, and how do we measure it? How are material properties estimated and represented? Resolving these questions causes us to scrutinize the basic assumptions of mid-level vision.

"What Not" Detectors Help the Brain See in Depth

Binocular stereopsis is one of the primary cues for three-dimensional (3D) vision in species ranging from insects to primates. Understanding how the brain extracts depth from two different retinal images represents a tractable challenge in sensory neuroscience that has so far evaded full explanation. Central to current thinking is the idea that the brain needs to identify matching features in the two retinal images (i.e., solving the “stereoscopic correspondence problem”) so that the depth of objects in the world can be triangulated. Although intuitive, this approach fails to account for key physiological and perceptual observations. We show that formulating the problem to identify “correct matches” is suboptimal and propose an alternative, based on optimal information encoding, that mixes disparity detection with “proscription”: exploiting dissimilar features to provide evidence against unlikely interpretations. We demonstrate the role of these “what not” responses in a neural network optimized to extract depth in natural images. The network combines information for and against the likely depth structure of the viewed scene, naturally reproducing key characteristics of both neural responses and perceptual interpretations. We capture the encoding and readout computations of the network in simple analytical form and derive a binocular likelihood model that provides a unified account of long-standing puzzles in 3D vision at the physiological and perceptual levels. We suggest that marrying detection with proscription provides an effective coding strategy for sensory estimation that may be useful for diverse feature domains (e.g., motion) and multisensory integration.

•

The brain uses “what not” detectors to facilitate 3D vision

•

Binocular mismatches are used to drive suppression of incompatible depths

•

Proscription accounts for depth perception without binocular correspondence

•

A simple analytical model captures perceptual and neural responses

Image deformation as a cue to material category judgment

Human observers easily recognize complex natural phenomena, such as flowing water, which often generate highly chaotic dynamic arrays of light on the retina. It has not been clarified how the visual system discerns the source of a fluid flow. Here we show that the magnitude of image deformation caused by light refraction is a critical factor for the visual system to determine the perceptual category of fluid flows. Employing a physics engine, we created computer-rendered scenes of water and hot air flows. For each flow, we manipulated the rendering parameters (distortion factors and the index of refraction) that strongly influence the magnitude of image deformation. The observers rated how strongly they felt impressions of water and hot air in the video clips of the flows. The ratings showed that the water and hot air impressions were positively and negatively related to the magnitude of image deformation. Based on the results, we discuss how the visual system heuristically utilizes image deformation to discern non-rigid materials such as water and hot air flows.

Lightness perception for matte and glossy complex shapes

Humans are able to estimate the reflective properties of the surface (albedo) of an object despite the large variability in the reflected light due to shading, illumination and specular reflection. Here we first used a physically based rendering simulation to study how different statistics (i.e. percentiles) based on the luminance distributions of matte and glossy objects predict the overall surface albedo. We found that the brightest parts of matte surfaces are good predictors of the surface albedo. As expected, the brightest parts led to poor performance in glossy surfaces. We then asked human observers to sort four (2 matte and 2 glossy) objects in a virtual scene in terms of their albedo. The brightest parts of matte surfaces highly correlated with human judgments, whereas in glossy surfaces, the highest correlation was achieved by percentiles within the darker half of the objects' luminance distributions. Furthermore, glossy surfaces tend to appear darker than matte ones, and observers are less precise in judging their lightness. We then manipulated different bands of the virtual objects' luminance distributions separately for glossy and matte surfaces. Modulating the brightest parts of the luminance distributions of the glossy surfaces had a limited impact on lightness perception, whereas it clearly influenced the perceived lightness of the matte objects. Our results demonstrate that human observers effectively ignore specular reflections while evaluating the lightness of glossy objects, which results in a bias to perceive glossy objects as darker.