Detection of light transformations and concomitant changes in surface albedo

How to make a #theDress

If we completely understand how a phenomenon works, we should be able to produce it ourselves. However, the individual differences in color appearance observed with #theDress seem to be a peculiarity of that photo, and it remains unclear how the proposed mechanisms underlying #theDress can be generalized to other images. Here, we developed a simple algorithm that transforms any image with bicolored objects into an image with the properties of #theDress. We measured the colors perceived in such images and compared them to those perceived in #theDress. Color adjustments confirmed that observers strongly differ in how they perceive the colors of the new images in a similar way as for #theDress. Most importantly, these differences were not unsystematic, but correlated with how observers perceive #theDress. These results imply that the color distribution is sufficient to produce the striking individual differences in color perception originally observed with #theDress-at least as long as the image appears realistic and hence compels the viewer to make assumptions about illuminations and surfaces. The algorithm can be used for stimulus production beyond this study.

Disentangling simultaneous changes of surface and illumination

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Motion Perception: From Detection to Interpretation

Visual motion processing can be conceptually divided into two levels. In the lower level, local motion signals are detected by spatiotemporal-frequency-selective sensors and then integrated into a motion vector flow. Although the model based on V1-MT physiology provides a good computational framework for this level of processing, it needs to be updated to fully explain psychophysical findings about motion perception, including complex motion signal interactions in the spatiotemporal-frequency and space domains. In the higher level, the velocity map is interpreted. Although there are many motion interpretation processes, we highlight the recent progress in research on the perception of material (e.g., specular reflection, liquid viscosity) and on animacy perception. We then consider possible linking mechanisms of the two levels and propose intrinsic flow decomposition as the key problem. To provide insights into computational mechanisms of motion perception, in addition to psychophysics and neurosciences, we review machine vision studies seeking to solve similar problems.

Accuracy and speed of material categorization in real-world images

It is easy to visually distinguish a ceramic knife from one made of steel, a leather jacket from one made of denim, and a plush toy from one made of plastic. Most studies of material appearance have focused on the estimation of specific material properties such as albedo or surface gloss, and as a consequence, almost nothing is known about how we recognize material categories like leather or plastic. We have studied judgments of high-level material categories with a diverse set of real-world photographs, and we have shown (Sharan, 2009) that observers can categorize materials reliably and quickly. Performance on our tasks cannot be explained by simple differences in color, surface shape, or texture. Nor can the results be explained by observers merely performing shape-based object recognition. Rather, we argue that fast and accurate material categorization is a distinct, basic ability of the visual system.

Inferred motion perception of light sources in 3D scenes is color-blind

In everyday scenes, the illuminant can vary spatially in chromaticity and luminance, and change over time (e.g. sunset). Such variation generates dramatic image effects too complex for any contemporary machine vision system to overcome, yet human observers are remarkably successful at inferring object properties separately from lighting, an ability linked with estimation and tracking of light field parameters. Which information does the visual system use to infer light field dynamics? Here, we specifically ask whether color contributes to inferred light source motion. Observers viewed 3D surfaces illuminated by an out-of-view moving collimated source (sun) and a diffuse source (sky). In half of the trials, the two sources differed in chromaticity, thereby providing more information about motion direction. Observers discriminated light motion direction above chance, and only the least sensitive observer benefited slightly from the added color information, suggesting that color plays only a very minor role for inferring light field dynamics.

Enhancement of glossiness perception by retinal-image motion: additional effect of head-yoked motion parallax

It has been argued that when an observer moves, a contingent retinal-image motion of a stimulus would strengthen the perceived glossiness. This would be attributed to the veridical perception of three-dimensional structure by motion parallax. However, it has not been investigated whether the effect of motion parallax is more than that of retinal-image motion of the stimulus. Using a magnitude estimation method, we examine in this paper whether cross-modal coordination of the stimulus change and the observer's motion (i.e., motion parallax) is essential or the retinal-image motion alone is sufficient for enhancing the perceived glossiness. Our data show that a retinal-image motion simulating motion parallax without head motion strengthened the perceived glossiness but that its effect was weaker than that of motion parallax with head motion. These results suggest the existence of an additional effect of the cross-modal coordination between vision and proprioception on glossiness perception. That is, motion parallax enhances the perception of glossiness, in addition to retinal-image motions of specular surfaces.

Learning to use illumination gradients as an unambiguous cue to three dimensional shape

The luminance and colour gradients across an image are the result of complex interactions between object shape, material and illumination. Using such variations to infer object shape or surface colour is therefore a difficult problem for the visual system. We know that changes to the shape of an object can affect its perceived colour, and that shading gradients confer a sense of shape. Here we investigate if the visual system is able to effectively utilise these gradients as a cue to shape perception, even when additional cues are not available. We tested shape perception of a folded card object that contained illumination gradients in the form of shading and more subtle effects such as inter-reflections. Our results suggest that observers are able to use the gradients to make consistent shape judgements. In order to do this, observers must be given the opportunity to learn suitable assumptions about the lighting and scene. Using a variety of different training conditions, we demonstrate that learning can occur quickly and requires only coarse information. We also establish that learning does not deliver a trivial mapping between gradient and shape; rather learning leads to the acquisition of assumptions about lighting and scene parameters that subsequently allow for gradients to be used as a shape cue. The perceived shape is shown to be consistent for convex and concave versions of the object that exhibit very different shading, and also similar to that delivered by outline, a largely unrelated cue to shape. Overall our results indicate that, although gradients are less reliable than some other cues, the relationship between gradients and shape can be quickly assessed and the gradients therefore used effectively as a visual shape cue.

Detection of changes in luminance distributions

How well can observers detect the presence of a change in luminance distributions? Performance was measured in three experiments. Observers viewed pairs of grayscale images on a calibrated CRT display. Each image was a checkerboard. All luminances in one image of each pair consisted of random draws from a single probability distribution. For the other image, some patch luminances consisted of random draws from that same distribution, while the rest of the patch luminances (test patches) consisted of random draws from a second distribution. The observers' task was to pick the image with luminances drawn from two distributions. The parameters of the second distribution that led to 75% correct performance were determined across manipulations of (1) the number of test patches, (2) the observers' certainty about test patch location, and (3) the geometric structure of the images. Performance improved with number of test patches and location certainty. The geometric manipulations did not affect performance. An ideal observer model with high efficiency fit the data well and a classification image analysis showed a similar use of information by the ideal and human observers, indicating that observers can make effective use of photometric information in our distribution discrimination task.

Surface color perception and equivalent illumination models

Vision provides information about the properties and identity of objects. The ease with which we perceive object properties belies the difficulty of the underlying information-processing task. In the case of object color, retinal information about object reflectance is confounded with information about the illumination as well as about the object's shape and pose. There is no obvious rule that allows transformation of the retinal image to a color representation that depends primarily on object surface reflectance. Under many circumstances, however, object color appearance is remarkably stable across scenes in which the object is viewed. Here, we review a line of experiments and theory that aim to understand how the visual system stabilizes object color appearance. Our emphasis is on models derived from explicit analysis of the computational problem of estimating the physical properties of illuminants and surfaces from the retinal image, and experiments that test these models. We argue that this approach has considerable promise for allowing generalization from simplified laboratory experiments to richer scenes that more closely approximate natural viewing. We discuss the relation between the work we review and other theoretical approaches available in the literature.

Color and material perception: achievements and challenges

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

Estimating changes in lighting direction in binocularly viewed three-dimensional scenes

We examine human ability to detect changes in scene lighting. Thirteen observers viewed three-dimensional rendered scenes stereoscopically. Each scene consisted of a randomly generated three-dimensional "Gaussian bump" surface rendered under a combination of collimated and diffuse light sources. During each trial, the collimated source underwent a small, quick change of position in one of four directions. The observer's task was to classify the direction of the lighting change. All observers were above chance in performing the task. We developed a model that combined two sources of information, a shape map and a shading map, to predict lighting change direction. We used this model to predict patterns of errors both across observers and across scenes differing in shape. We found that errors in estimating lighting direction were primarily the result of errors in representing surface shape. We characterized the surface features that affected performance in the classification task.