Surface color perception in three-dimensional scenes

Distinct processes of lighting priors for lightness and 3-D shape perception

The visual system often relies on prior assumptions when interpreting ambiguous visual inputs. A well-known example is the light-from-above prior, which aids the judgment of an object's three-dimensional (3-D) shape (i.e., convex or concave). Recent studies have revealed that the light-from-above prior also helps solve lightness ambiguity. This study aimed to examine whether 3-D shape perception and lightness perception share the same lighting prior. The study participants performed two tasks: one focusing on lightness perception and another focusing on 3-D shape perception. The dominant directions of the assumed lighting were calculated from participants’ performance in the two tasks. The results showed that the assumed lighting direction for 3-D shape perception were considerably biased toward the left, whereas the one for lightness perception was almost from directly above. The clear difference between these two directions supports the hypothesis that the visual system uses distinct lighting priors for 3-D shape perception and lightness perception. Experiments 1 and 2 involved Japanese speaking participants and European participants, respectively. The Japanese language can be read and written both horizontally (i.e., left to right) and vertically (i.e., up to down) with lines progressing from right to left. Nevertheless, the two experiments still produced the same result, which suggests that the present finding is universal regardless of reading/writing direction.

Surface properties and the perception of color

We examined whether perception of color saturation and lightness depends on the three-dimensional (3D) shape and surface gloss of surfaces rendered to have different hues. In Experiment 1, we parametrically varied specular roughness of predominantly planar surfaces with different mesoscopic relief heights. The orientation of surfaces was varied relative to the light source and observer. Observers matched perceived lightness and chroma (effectively saturation) using spherical objects rendered using CIE LCH color space. We observed strong interactions between perceived saturation and lightness with changes in surface orientation and surface properties (specular roughness and 3D relief height). Declines in saturation and increases in lightness were observed with increasing specular roughness. Changes in relief height had greater effects on perceived saturation and lightness for blue hues compared with reddish and greenish hues. Experiment 2 found inverse correlations between perceived gloss and specular roughness across conditions. Experiment 3 estimated perceived specular coverage and found that a weighted combination of perceived gloss and specular coverage could account for perceived color saturation and lightness, with different coefficients accounting for the perceptual experience for each of the three hue conditions. These findings suggest that perceived color saturation and lightness depend on the separation of specular highlights from diffuse shading informative of chromatic surface reflectance.

Light: Toward a Transdisciplinary Science of Appearance and Atmosphere

To understand the processes behind seeing light, we need to integrate knowledge about the incoming optical structure, its perception, and how light interacts with material, shape, and space-objectively and subjectively. To that end, we need a novel approach to the science of light, namely, a transdisciplinary science of appearance, integrating optical, perceptual, and design knowledge and methods. In this article, I review existing literature as a basis for such a synthesis, which should discuss light in its full complexity, including its spatial properties and interactions with materials, shape, and space. I propose to investigate this by representing the endless variety of light, materials, shapes, and space as canonical modes and their combinations.

An Upward-Facing Surface Appears Darker: The Role Played by the Light-From-Above Assumption in Lightness Perception

The human visual system can extract information on surface reflectance (lightness) from light intensity; this, however, confounds information on reflectance and illumination. We hypothesized that the visual system, to solve this lightness problem, utilizes the internally held prior assumption that illumination falls from above. Experiment 1 showed that an upward-facing surface is perceived to be darker than a downward-facing surface, proving our hypothesis. Experiment 2 showed the same results in the absence of explicit illumination cues. The effect of the light-from-left prior assumption was not observed in Experiment 3. The upward- and downward-facing surface stimuli in Experiments 1 and 2 showed no difference in a two-dimensional configuration or three-dimensional structure, and the participants' perceived lightness appeared to be affected by the observers' prior assumption that illumination is always from above. Other studies have not accounted for this illusory effect, and this study's finding provides additional insights into the study of lightness perception.

Disentangling simultaneous changes of surface and illumination

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Color Constancy in Two-Dimensional and Three-Dimensional Scenes: Effects of Viewing Methods and Surface Texture

There has been debate about how and why color constancy may be better in three-dimensional (3-D) scenes than in two-dimensional (2-D) scenes. Although some studies have shown better color constancy for 3-D conditions, the role of specific cues remains unclear. In this study, we compared color constancy for a 3-D miniature room (a real scene consisting of actual objects) and 2-D still images of that room presented on a monitor using three viewing methods: binocular viewing, monocular viewing, and head movement. We found that color constancy was better for the 3-D room; however, color constancy for the 2-D image improved when the viewing method caused the scene to be perceived more like a 3-D scene. Separate measurements of the perceptual 3-D effect of each viewing method also supported these results. An additional experiment comparing a miniature room and its image with and without texture suggested that surface texture of scene objects contributes to color constancy.

Context-dependent lightness affects perceived contrast

Perceived contrast of a grating varies with its background (or mean) luminance: of the two gratings with the same photometric contrast the one on higher luminance background appears to have higher contrast. Does perceived contrast also vary with context-dependent background lightness even when the luminance remains constant? We investigated this question using a stimulus in which two equiluminant patches ("context squares", CSs) appear different in lightness. First we measured the lightness effect in a behavioral experiment. After ensuring that it was present for all participants, we conducted perceived contrast experiments, where participants judged the contrast of rectified incremental and decremental square-wave gratings superimposed on the CSs. For the incremental gratings participants' settings were significantly different for the two CSs. Specifically, perceived contrast was higher when the gratings were placed on the context square that was perceived lighter. In a follow-up experiment we measured perceived contrast of rectified gratings on isolated patches that differed in luminance. The pattern of results of the two experiments was consistent, demonstrating that possibly shared mechanisms underpin the effects of background luminance and context-dependent lightness on perceived contrast.

Seeing “the Dress” in the Right Light: Perceived Colors and Inferred Light Sources

In the well-known “dress” photograph, people either see the dress as blue with black stripes or as white with golden stripes. We suggest that the perception of colors is guided by the scene interpretation and the inferred positions of light sources. We tested this hypothesis in two online studies using color matching to estimate the colors observers see, while controlling for individual differences in gray point bias and color discrimination. Study 1 demonstrates that the interpretation of the dress corresponds to differences in perceived colors. Moreover, people who perceive the dress as blue-and-black are two times more likely to consider the light source as frontal, than those who see the white-and-gold dress. The inferred light sources, in turn, depend on the circadian changes in ambient light. The interpretation of the scene background as a wall or a mirror is consistent with the perceived colors as well. Study 2 shows that matching provides reliable results on differing devices and replicates the findings on scene interpretation and light sources. Additionally, we show that participants’ environmental lighting conditions are an important cue for perceiving the dress colors. The exact mechanisms of how environmental lighting and circadian changes influence the perceived colors of the dress deserve further investigation.

Low levels of specularity support operational color constancy, particularly when surface and illumination geometry can be inferred

We tested whether surface specularity alone supports operational color constancy-the ability to discriminate changes in illumination or reflectance. Observers viewed short animations of illuminant or reflectance changes in rendered scenes containing a single spherical surface and were asked to classify the change. Performance improved with increasing specularity, as predicted from regularities in chromatic statistics. Peak performance was impaired by spatial rearrangements of image pixels that disrupted the perception of illuminated surfaces but was maintained with increased surface complexity. The characteristic chromatic transformations that are available with nonzero specularity are useful for operational color constancy, particularly if accompanied by appropriate perceptual organization.

Shape, Illumination, and Reflectance from Shading

A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison-there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint tends to be uniform, and illumination tends to be natural. We therefore pose this problem as one of statistical inference, and define an optimization problem that searches for the most likely explanation of a single image. Our technique can be viewed as a superset of several classic computer vision problems (shape-from-shading, intrinsic images, color constancy, illumination estimation, etc) and outperforms all previous solutions to those constituent problems.

The effect of background and illumination on color identification of real, 3D objects

For the surface reflectance of an object to be a useful cue to object identity, judgments of its color should remain stable across changes in the object's environment. In 2D scenes, there is general consensus that color judgments are much more stable across illumination changes than background changes. Here we investigate whether these findings generalize to real 3D objects. Observers made color matches to cubes as we independently varied both the illumination impinging on the cube and the 3D background of the cube. As in 2D scenes, we found relatively high but imperfect stability of color judgments under an illuminant shift. In contrast to 2D scenes, we found that background had little effect on average color judgments. In addition, variability of color judgments was increased by an illuminant shift and decreased by embedding the cube within a background. Taken together, these results suggest that in real 3D scenes with ample cues to object segregation, the addition of a background may improve stability of color identification.

We infer light in space

In studies of lightness and color constancy, the terms lightness and brightness refer to the qualia corresponding to perceived surface reflectance and perceived luminance, respectively. However, what has rarely been considered is the fact that the volume of space containing surfaces appears neither empty, void, nor black, but filled with light. Helmholtz (1866/1962) came closest to describing this phenomenon when discussing inferred illumination, but previous theoretical treatments have fallen short by restricting their considerations to the surfaces of objects. The present work is among the first to explore how we infer the light present in empty space. It concludes with several research examples supporting the theory that humans can infer the differential levels and chromaticities of illumination in three-dimensional space.

Learning to use illumination gradients as an unambiguous cue to three dimensional shape

The luminance and colour gradients across an image are the result of complex interactions between object shape, material and illumination. Using such variations to infer object shape or surface colour is therefore a difficult problem for the visual system. We know that changes to the shape of an object can affect its perceived colour, and that shading gradients confer a sense of shape. Here we investigate if the visual system is able to effectively utilise these gradients as a cue to shape perception, even when additional cues are not available. We tested shape perception of a folded card object that contained illumination gradients in the form of shading and more subtle effects such as inter-reflections. Our results suggest that observers are able to use the gradients to make consistent shape judgements. In order to do this, observers must be given the opportunity to learn suitable assumptions about the lighting and scene. Using a variety of different training conditions, we demonstrate that learning can occur quickly and requires only coarse information. We also establish that learning does not deliver a trivial mapping between gradient and shape; rather learning leads to the acquisition of assumptions about lighting and scene parameters that subsequently allow for gradients to be used as a shape cue. The perceived shape is shown to be consistent for convex and concave versions of the object that exhibit very different shading, and also similar to that delivered by outline, a largely unrelated cue to shape. Overall our results indicate that, although gradients are less reliable than some other cues, the relationship between gradients and shape can be quickly assessed and the gradients therefore used effectively as a visual shape cue.

Contributions of ideal observer theory to vision research

An ideal observer is a hypothetical device that performs optimally in a perceptual task given the available information. The theory of ideal observers has proven to be a powerful and useful tool in vision research, which has been applied to a wide range of problems. Here I first summarize the basic concepts and logic of ideal observer analysis and then briefly describe applications in a number of different areas, including pattern detection, discrimination and estimation, perceptual grouping, shape, depth and motion perception and visual attention, with an emphasis on recent applications. Given recent advances in mathematical statistics, in computational power, and in techniques for measuring behavioral performance, neural activity and natural scene statistics, it seems certain that ideal observer theory will play an ever increasing role in basic and applied areas of vision science.

Color and material perception: achievements and challenges

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

Color constancy

A quarter of a century ago, the first systematic behavioral experiments were performed to clarify the nature of color constancy-the effect whereby the perceived color of a surface remains constant despite changes in the spectrum of the illumination. At about the same time, new models of color constancy appeared, along with physiological data on cortical mechanisms and photographic colorimetric measurements of natural scenes. Since then, as this review shows, there have been many advances. The theoretical requirements for constancy have been better delineated and the range of experimental techniques has been greatly expanded; novel invariant properties of images and a variety of neural mechanisms have been identified; and increasing recognition has been given to the relevance of natural surfaces and scenes as laboratory stimuli. Even so, there remain many theoretical and experimental challenges, not least to develop an account of color constancy that goes beyond deterministic and relatively simple laboratory stimuli and instead deals with the intrinsically variable nature of surfaces and illuminations present in the natural world.

Detection of light transformations and concomitant changes in surface albedo

We report two experiments demonstrating that (1) observers are sensitive to information about changes in the light field not captured by local scene statistics and that (2) they can use this information to enhance detection of changes in surface albedo. Observers viewed scenes consisting of matte surfaces at many orientations illuminated by a collimated light source. All surfaces were achromatic, all lights neutral. In the first experiment, observers attempted to discriminate small changes in direction of the collimated light source (light transformations) from matched changes in the albedos of all surfaces (non-light transformations). Light changes and non-light changes shared the same local scene statistics and edge ratios, but the latter were not consistent with any change in direction to the collimated source. We found that observers could discriminate light changes as small as 5 degrees with sensitivity d' > 1 and accurately judge the direction of change. In a second experiment, we measured observers' ability to detect a change in the surface albedo of an isolated surface patch during either a light change or a surface change. Observers were more accurate in detecting isolated albedo changes during light changes. Measures of sensitivity d' were more than twice as great.

Contrast, constancy, and measurements of perceived lightness under parametric manipulation of surface slant and surface reflectance

Across many scenes, local contrast provides a valid cue to surface reflectance, but it is not the only such cue. To generalize beyond theories of lightness that rely exclusively on local contrast, we need to know which other potential cues matter. We had observers make lightness matches between two scene locations, and varied the surface slant and local surround reflectance of one of the locations. When local contrast was a valid cue to reflectance, all observers were approximately lightness constant. When it was not, observers' lightness matches were intermediate between contrast matching and lightness constancy. For most observers, surface slant exerted an effect on perceived lightness beyond that explainable by local contrast.

Effect of pictorial depth cues, binocular disparity cues and motion parallax depth cues on lightness perception in three-dimensional virtual scenes

Background

Surface lightness perception is affected by scene interpretation. There is some experimental evidence that perceived lightness under bi-ocular viewing conditions is different from perceived lightness in actual scenes but there are also reports that viewing conditions have little or no effect on perceived color. We investigated how mixes of depth cues affect perception of lightness in three-dimensional rendered scenes containing strong gradients of illumination in depth.

Methodology/Principal Findings

Observers viewed a virtual room (4 m width×5 m height×17.5 m depth) with checkerboard walls and floor. In four conditions, the room was presented with or without binocular disparity (BD) depth cues and with or without motion parallax (MP) depth cues. In all conditions, observers were asked to adjust the luminance of a comparison surface to match the lightness of test surfaces placed at seven different depths (8.5–17.5 m) in the scene. We estimated lightness versus depth profiles in all four depth cue conditions. Even when observers had only pictorial depth cues (no MP, no BD), they partially but significantly discounted the illumination gradient in judging lightness. Adding either MP or BD led to significantly greater discounting and both cues together produced the greatest discounting. The effects of MP and BD were approximately additive. BD had greater influence at near distances than far.

Conclusions/Significance

These results suggest the surface lightness perception is modulated by three-dimensional perception/interpretation using pictorial, binocular-disparity, and motion-parallax cues additively. We propose a two-stage (2D and 3D) processing model for lightness perception.

Testing limits on matte surface color perception in three-dimensional scenes with complex light fields

We investigated limits on the human visual system's ability to discount directional variation in complex lights field when estimating Lambertian surface color. Directional variation in the light field was represented in the frequency domain using spherical harmonics. The bidirectional reflectance distribution function of a Lambertian surface acts as a low-pass filter on directional variation in the light field. Consequently, the visual system needs to discount only the low-pass component of the incident light corresponding to the first nine terms of a spherical harmonics expansion [Basri, R., Jacobs, D. (2001). Lambertian reflectance and linear subspaces. In: International Conference on Computer Vision II, pp. 383-390; Ramamoorthi, R., Hanrahan, P., (2001). An efficient representation for irradiance environment maps. SIGGRAPH 01. New York: ACM Press, pp. 497-500] to accurately estimate surface color. We test experimentally whether the visual system discounts directional variation in the light field up to this physical limit. Our results are consistent with the claim that the visual system can compensate for all of the complexity in the light field that affects the appearance of Lambertian surfaces.

Visual Perception and the Statistical Properties of Natural Scenes

The environments in which we live and the tasks we must perform to survive and reproduce have shaped the design of our perceptual systems through evolution and experience. Therefore, direct measurement of the statistical regularities in natural environments (scenes) has great potential value for advancing our understanding of visual perception. This review begins with a general discussion of the natural scene statistics approach, of the different kinds of statistics that can be measured, and of some existing measurement techniques. This is followed by a summary of the natural scene statistics measured over the past 20 years. Finally, there is a summary of the hypotheses, models, and experiments that have emerged from the analysis of natural scene statistics.

Responses to lightness variations in early human visual cortex

Lightness is the apparent reflectance of a surface, and it depends not only on the actual luminance of the surface but also on the context in which the surface is viewed [1-10]. The cortical mechanisms of lightness processing are largely unknown, and the role of early cortical areas is still a matter of debate [11-17]. We studied the cortical responses to lightness variations in early stages of the human visual system with functional magnetic resonance imaging (fMRI) while observers were performing a demanding fixation task. The set of dynamically presented visual stimuli included the rectangular version of the classic Craik-O'Brien stimulus [3, 18, 19] and a variant that led to a weaker lightness effect, as well as a pattern with actual luminance variations. We found that the cortical activity in retinotopic areas, including the primary visual cortex (V1), is correlated with context-dependent lightness variations.