Illuminant cues in surface color perception: tests of three candidate cues

Transparent layer constancy improves with increased naturalness of the scene

The extent to which hue, saturation, and transmittance of thin light-transmitting layers are perceived as constant when the illumination changes (transparent layer constancy, TLC) has previously been investigated with simple stimuli in asymmetric matching tasks. In this task, a target filter is presented under one illumination and a second filter is matched under a second illumination. Although two different illuminations are applied in the stimulus generation, there is no guarantee that the stimulus will be interpreted appropriately by the visual system. In previous work, we found a higher degree of TLC when both illuminations were presented alternately than when they were presented simultaneously, which could be explained, for example, by an increased plausibility of an illumination change. In this work, we test whether TLC can also be increased in simultaneous presentation when the filter's belonging to a particular illumination context is made more likely by additional cues. To this end, we presented filters in differently lit areas of complex, naturalistically rendered 3D scenes containing different types of cues to the prevailing illumination, such as scene geometry, object shading, and cast shadows. We found higher degrees of TLC in such complex scenes than in colorimetrically similar simple 2D color mosaics, which is consistent with the results of similar studies in the area of color constancy. To test which of the illumination cues available in the scenes are actually used, the different types of cues were successively removed from the naturalistically rendered complex scene. A total of eight levels of scene complexity were examined. As expected, TLC decreased the more cues were removed. Object shading and illumination gradients due to shadow cast were both found to have a positive effect on TLC. A second filter had a small positive effect on TLC when added in strongly reduced scenes, but not in the complex scenes that already provide many cues about the illumination context of the filter.

Colour expectations across illumination changes

This study investigates human expectations towards naturalistic colour changes under varying illuminations. Understanding colour expectations is key to both scientific research on colour constancy and applications of colour and lighting in art and industry. We reanalysed data from asymmetric colour matches of a previous study and found that colour adjustments tended to align with illuminant-induced colour shifts predicted by naturalistic, rather than artificial, illuminants and reflectances. We conducted three experiments using hyperspectral images of naturalistic scenes to test if participants judged colour changes based on naturalistic illuminant and reflectance spectra as more plausible than artificial ones, which contradicted their expectations. When we consistently manipulated the illuminant (Experiment 1) and reflectance (Experiment 2) spectra across the whole scene, observers chose the naturalistic renderings significantly above the chance level (>25 %) but barely more often than any of the three artificial ones, collectively (>50 %). However, when we manipulated only one object/area's reflectance (Experiment 3), observers more reliably identified the version in which the object had a naturalistic reflectance like the rest of the scene. Results from Experiments 2-3 and additional analyses suggested that relational colour constancy strongly contributed to observer expectations, and stable cone-excitation ratios are not limited to naturalistic illuminants and reflectances but also occur for our artificial renderings. Our findings indicate that relational colour constancy and prior knowledge about surface colour shifts help to disambiguate surface colour identity under illumination changes, enabling human observers to recognise surface colours reliably in naturalistic conditions. Additionally, relational colour constancy may even be effective in many artificial conditions.

The Reality of a Head-Mounted Display (HMD) Environment Tested via Lightness Perception

Head-mounted displays (HMDs) are becoming more and more popular as a device for displaying a virtual reality space, but how real are they? The present study attempted to quantitatively evaluate the degree of reality achieved with HMDs by using a perceptual phenomenon as a measure. Lightness constancy is an ability that is present in human visual perception, in which the perceived reflectance (i.e., the lightness) of objects appears to stay constant across illuminant changes. Studies on color/lightness constancy in humans have shown that the degree of constancy is high, in general, when real objects are used as stimuli. We asked participants to make lightness matches between two virtual environments with different illuminant intensities, as presented in an HMD. The participants' matches showed a high degree of lightness constancy in the HMD; our results marked no less than 74.2% (84.8% at the maximum) in terms of the constancy index, whereas the average score on the computer screen was around 65%. The effect of head-tracking ability was confirmed by disabling that function, and the result showed a significant drop in the constancy index but that it was equally effective when the virtual environment was generated by replay motions. HMDs yield a realistic environment, with the extension of the visual scene being accompanied by head motions.

Color and gloss constancy under diverse lighting environments

When we look at an object, we simultaneously see how glossy or matte it is, how light or dark, and what color. Yet, at each point on the object's surface, both diffuse and specular reflections are mixed in different proportions, resulting in substantial spatial chromatic and luminance variations. To further complicate matters, this pattern changes radically when the object is viewed under different lighting conditions. The purpose of this study was to simultaneously measure our ability to judge color and gloss using an image set capturing diverse object and illuminant properties. Participants adjusted the hue, lightness, chroma, and specular reflectance of a reference object so that it appeared to be made of the same material as a test object. Critically, the two objects were presented under different lighting environments. We found that hue matches were highly accurate, except for under a chromatically atypical illuminant. Chroma and lightness constancy were generally poor, but these failures correlated well with simple image statistics. Gloss constancy was particularly poor, and these failures were only partially explained by reflection contrast. Importantly, across all measures, participants were highly consistent with one another in their deviations from constancy. Although color and gloss constancy hold well in simple conditions, the variety of lighting and shape in the real world presents significant challenges to our visual system's ability to judge intrinsic material properties.

The dominating impacts of Helmholtz-Kohlrausch effect on color-induced glossiness enhancement

Glossiness can be increased by adding chromatic information to the object images. However, the mechanisms that create color-induced glossiness enhancement are unclear. In this study, we psychophysically measured the glossiness of object images to which various hue chromaticities were added to elucidate the perceptual and image factors that explain the color-induced glossiness enhancement effect. Two types of coloring conditions were tested: the both-colored (BC) condition, in which both specular and diffuse components were colored with the same chromaticity, and the diffuse-colored (DC) condition, in which only diffuse components were colored while specular components remained achromatic. The results showed that glossiness enhancement was more prominent in the BC than in the DC condition, and the dependency of glossiness enhancement on the stimulus color direction was similar to that of the Helmholtz-Kohlrausch (H-K) effect. Furthermore, we performed a regression analysis with a linear mixed model based on image features and an additional experiment in which an H-K effect-based increase in perceived brightness was imitated on achromatic stimuli by manipulating luminance. The results demonstrated that the H-K effect-based brightness enhancement in the highlight regions explains the glossiness enhancement effect well. These results suggest that the H-K effect, especially around the highlight region, is a dominant factor that creates the color-induced glossiness enhancement, although other color-related factors could also be partly involved.

Equivalent noise characterization of human lightness constancy

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers’ internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest – within a factor of two of the equivalent noise level set by internal noise.

Deep neural models for color classification and color constancy

Color constancy is our ability to perceive constant colors across varying illuminations. Here, we trained deep neural networks to be color constant and evaluated their performance with varying cues. Inputs to the networks consisted of two-dimensional images of simulated cone excitations derived from three-dimensional (3D) rendered scenes of 2,115 different 3D shapes, with spectral reflectances of 1,600 different Munsell chips, illuminated under 278 different natural illuminations. The models were trained to classify the reflectance of the objects. Testing was done with four new illuminations with equally spaced CIEL*a*b* chromaticities, two along the daylight locus and two orthogonal to it. High levels of color constancy were achieved with different deep neural networks, and constancy was higher along the daylight locus. When gradually removing cues from the scene, constancy decreased. Both ResNets and classical ConvNets of varying degrees of complexity performed well. However, DeepCC, our simplest sequential convolutional network, represented colors along the three color dimensions of human color vision, while ResNets showed a more complex representation.

The Certainty of Ambiguity in Visual Neural Representations

Some images evoke bistable percepts: two different visual experiences seen in alternation while continuously viewing an unchanged stimulus. The Necker Cube and Rubin's Vase are classic examples, each of which gives alternating percepts of different shapes. Other bistable percepts are alternating colors or directions of motion. Although stimuli that result in salient bistability are rare and sometimes cleverly constructed to emphasize ambiguity, they have been influential for over 150 years, since the work of von Helmholtz, who considered them to be evidence for perceptual visual processes that interpret retinal stimuli. While bistability in natural viewing is uncommon, the main point of this review is that implicit ambiguity in visual neural representations is pervasive. Resolving ambiguity, therefore, is a fundamental and ubiquitous process of vision that routinely affects what we see, not an oddity arising from cleverly crafted images. This review focuses on the causes of widespread ambiguity, historical perspectives on it, and modern knowledge and theory about resolving it.

Surface properties and the perception of color

We examined whether perception of color saturation and lightness depends on the three-dimensional (3D) shape and surface gloss of surfaces rendered to have different hues. In Experiment 1, we parametrically varied specular roughness of predominantly planar surfaces with different mesoscopic relief heights. The orientation of surfaces was varied relative to the light source and observer. Observers matched perceived lightness and chroma (effectively saturation) using spherical objects rendered using CIE LCH color space. We observed strong interactions between perceived saturation and lightness with changes in surface orientation and surface properties (specular roughness and 3D relief height). Declines in saturation and increases in lightness were observed with increasing specular roughness. Changes in relief height had greater effects on perceived saturation and lightness for blue hues compared with reddish and greenish hues. Experiment 2 found inverse correlations between perceived gloss and specular roughness across conditions. Experiment 3 estimated perceived specular coverage and found that a weighted combination of perceived gloss and specular coverage could account for perceived color saturation and lightness, with different coefficients accounting for the perceptual experience for each of the three hue conditions. These findings suggest that perceived color saturation and lightness depend on the separation of specular highlights from diffuse shading informative of chromatic surface reflectance.

Specular highlights improve color constancy when other cues are weakened

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination ("daylight priors"). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy. Observers made achromatic settings within three-dimensional rendered scenes containing matte or glossy shapes, illuminated by either daylight or nondaylight illuminations. We assessed both the variability of these settings and their accuracy, in terms of the standard color constancy index (CCI). When a spectrally uniform background was present, neither CCIs nor variability improved with specular highlights or daylight illuminants (Experiment 1). When a Mondrian background was introduced, CCIs decreased overall but were higher for scenes containing glossy, as opposed to matte, shapes (Experiments 2 and 3). There was no overall reduction in variability of settings and no benefit for scenes illuminated by daylights. Taken together, these results suggest that the human visual system indeed uses specular highlights to improve color constancy but only when other cues, such as from the local surround, are weakened.

Disentangling simultaneous changes of surface and illumination

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Illumination discrimination for chromatically biased illuminations: Implications for color constancy

We measured discrimination thresholds for illumination changes along different chromatic directions starting from chromatically biased reference illuminations. Participants viewed a Mondrian-papered scene illuminated by LED lamps. The scene was first illuminated by a reference illumination, followed by two comparisons. One comparison matched the reference (the target); the other (the test) varied from the reference, nominally either bluer, yellower, redder, or greener. The participant's task was to correctly select the target. A staircase procedure found thresholds for discrimination of an illumination change along each axis of chromatic change. Nine participants completed the task for five different reference illumination conditions (neutral, blue, yellow, red, and green). We find that relative discrimination thresholds for different chromatic directions of illumination change vary with the reference illumination. For the neutral reference, there is a trend for thresholds to be highest in the bluer illumination-change direction, replicating our previous reports of a "blue bias" for neutral reference illuminations. For the four chromatic references (blue, yellow, red, and green), the change in illumination toward the neutral reference is less well discriminated than changes in the other directions: a "neutral bias." The results have implications for color constancy: In considering the stability of surface appearance under changes in illumination, both the starting chromaticity of the illumination and direction of change must be considered, as well as the chromatic characteristics of the surface reflectance ensemble. They also suggest it will be worthwhile to explore whether and how the human visual system has internalized the statistics of natural illumination changes.

Computational luminance constancy from naturalistic images

The human visual system supports stable percepts of object color even though the light that reflects from object surfaces varies significantly with the scene illumination. To understand the computations that support stable color perception, we study how estimating a target object's luminous reflectance factor (LRF; a measure of the light reflected from the object under a standard illuminant) depends on variation in key properties of naturalistic scenes. Specifically, we study how variation in target object reflectance, illumination spectra, and the reflectance of background objects in a scene impact estimation of a target object's LRF. To do this, we applied supervised statistical learning methods to the simulated excitations of human cone photoreceptors, obtained from labeled naturalistic images. The naturalistic images were rendered with computer graphics. The illumination spectra of the light sources and the reflectance spectra of the surfaces in the scene were generated using statistical models of natural spectral variation. Optimally decoding target object LRF from the responses of a small learned set of task-specific linear receptive fields that operate on a contrast representation of the cone excitations yields estimates that are within 13% of the correct LRF. Our work provides a framework for evaluating how different sources of scene variability limit performance on luminance constancy.

Color Perception: Objects, Constancy, and Categories

Color has been scientifically investigated by linking color appearance to colorimetric measurements of the light that enters the eye. However, the main purpose of color perception is not to determine the properties of incident light, but to aid the visual perception of objects and materials in our environment. We review the state of the art on object colors, color constancy, and color categories to gain insight into the functional aspects of color perception. The common ground between these areas of research is that color appearance is tightly linked to the identification of objects and materials and the communication across observers. In conclusion, we argue that research should focus on how color processing is adapted to the surface properties of objects in the natural environment in order to bridge the gap between the known early stages of color perception and the subjective appearance of color.

Discrimination of spectral reflectance under environmental illumination

Color constancy is the ability to recover a stable perceptual estimate of surface reflectance, regardless of the lighting environment. However, we know little about how observers make judgments of the surface color of glossy objects, particularly in complex lighting environments that introduce complex spatial patterns of chromatic variation across an object's surface. To address this question, we measured thresholds for reflectance discrimination using computer-rendered stimuli under environmental illumination. In Experiment 1, we found that glossiness and shape had small effects on discrimination thresholds. Importantly, discrimination ellipses extended along the direction in which the chromaticities in the environmental illumination spread. In Experiment 2, we also found that the observers' abilities to judge surface colors were worse in lighting environments with an atypical chromatic distribution.

Colour Vision: Understanding #TheDress

A widely-viewed image of a dress elicits striking individual variation in colour perception. Experiments with multiple variants of the image suggest that the individual differences may arise through the action of visual mechanisms that normally stabilise object colour.

Low levels of specularity support operational color constancy, particularly when surface and illumination geometry can be inferred

We tested whether surface specularity alone supports operational color constancy-the ability to discriminate changes in illumination or reflectance. Observers viewed short animations of illuminant or reflectance changes in rendered scenes containing a single spherical surface and were asked to classify the change. Performance improved with increasing specularity, as predicted from regularities in chromatic statistics. Peak performance was impaired by spatial rearrangements of image pixels that disrupted the perception of illuminated surfaces but was maintained with increased surface complexity. The characteristic chromatic transformations that are available with nonzero specularity are useful for operational color constancy, particularly if accompanied by appropriate perceptual organization.

Material properties from contours: New insights on object perception

In this work we explored phenomenologically the visual complexity of the material attributes on the basis of the contours that define the boundaries of a visual object. The starting point is the rich and pioneering work done by Gestalt psychologists and, more in detail, by Rubin, who first demonstrated that contours contain most of the information related to object perception, like the shape, the color and the depth. In fact, by investigating simple conditions like those used by Gestalt psychologists, mostly consisting of contours only, we demonstrated that the phenomenal complexity of the material attributes emerges through appropriate manipulation of the contours. A phenomenological approach, analogous to the one used by Gestalt psychologists, was used to answer the following questions. What are contours? Which attributes can be phenomenally defined by contours? Are material properties determined only by contours? What is the visual syntactic organization of object attributes? The results of this work support the idea of a visual syntactic organization as a new kind of object formation process useful to understand the language of vision that creates well-formed attribute organizations. The syntax of visual attributes can be considered as a new way to investigate the modular coding and, more generally, the binding among attributes, i.e., the issue of how the brain represents the pairing of shape and material properties.

Accuracy and speed of material categorization in real-world images

It is easy to visually distinguish a ceramic knife from one made of steel, a leather jacket from one made of denim, and a plush toy from one made of plastic. Most studies of material appearance have focused on the estimation of specific material properties such as albedo or surface gloss, and as a consequence, almost nothing is known about how we recognize material categories like leather or plastic. We have studied judgments of high-level material categories with a diverse set of real-world photographs, and we have shown (Sharan, 2009) that observers can categorize materials reliably and quickly. Performance on our tasks cannot be explained by simple differences in color, surface shape, or texture. Nor can the results be explained by observers merely performing shape-based object recognition. Rather, we argue that fast and accurate material categorization is a distinct, basic ability of the visual system.

No Measured Effect of a Familiar Contextual Object on Color Constancy

Some familiar objects have a typical color, such as the yellow of a banana. The presence of such objects in a scene is a potential cue to the scene illumination, since the light reflected from them should on average be consistent with their typical surface reflectance. Although there are many studies on how the identity of an object affects how its color is perceived, little is known about whether the presence of a familiar object in a scene helps the visual system stabilize the color appearance of other objects with respect to changes in illumination. We used a successive color matching procedure in three experiments designed to address this question. Across the experiments we studied a total of 6 subjects (2 in Experiment 1, 3 in Experiment 2, and 4 in Experiment 3) with partial overlap of subjects between experiments. We compared measured color constancy across conditions in which a familiar object cue to the illuminant was available with conditions in which such a cue was not present. Overall, our results do not reveal a reliable improvement in color constancy with the addition of a familiar object to a scene. An analysis of the experimental power of our data suggests that if there is such an effect, it is small: less than approximately a change of 0.09 in a constancy index where an absence of constancy corresponds to an index value of 0 and perfect constancy corresponds to an index value of 1.

Color constancy in a scene with bright colors that do not have a fully natural surface appearance

Theoretical and experimental approaches have proposed that color constancy involves a correction related to some average of stimulation over the scene, and some of the studies showed that the average gives greater weight to surrounding bright colors. However, in a natural scene, high-luminance elements do not necessarily carry information about the scene illuminant when the luminance is too high for it to appear as a natural object color. The question is how a surrounding color's appearance mode influences its contribution to the degree of color constancy. Here the stimuli were simple geometric patterns, and the luminance of surrounding colors was tested over the range beyond the luminosity threshold. Observers performed perceptual achromatic setting on the test patch in order to measure the degree of color constancy and evaluated the surrounding bright colors' appearance mode. Broadly, our results support the assumption that the visual system counts only the colors in the object-color appearance for color constancy. However, detailed analysis indicated that surrounding colors without a fully natural object-color appearance had some sort of influence on color constancy. Consideration of this contribution of unnatural object color might be important for precise modeling of human color constancy.

Chromatic illumination discrimination ability reveals that human colour constancy is optimised for blue daylight illuminations

The phenomenon of colour constancy in human visual perception keeps surface colours constant, despite changes in their reflected light due to changing illumination. Although colour constancy has evolved under a constrained subset of illuminations, it is unknown whether its underlying mechanisms, thought to involve multiple components from retina to cortex, are optimised for particular environmental variations. Here we demonstrate a new method for investigating colour constancy using illumination matching in real scenes which, unlike previous methods using surface matching and simulated scenes, allows testing of multiple, real illuminations. We use real scenes consisting of solid familiar or unfamiliar objects against uniform or variegated backgrounds and compare discrimination performance for typical illuminations from the daylight chromaticity locus (approximately blue-yellow) and atypical spectra from an orthogonal locus (approximately red-green, at correlated colour temperature 6700 K), all produced in real time by a 10-channel LED illuminator. We find that discrimination of illumination changes is poorer along the daylight locus than the atypical locus, and is poorest particularly for bluer illumination changes, demonstrating conversely that surface colour constancy is best for blue daylight illuminations. Illumination discrimination is also enhanced, and therefore colour constancy diminished, for uniform backgrounds, irrespective of the object type. These results are not explained by statistical properties of the scene signal changes at the retinal level. We conclude that high-level mechanisms of colour constancy are biased for the blue daylight illuminations and variegated backgrounds to which the human visual system has typically been exposed.

RenderToolbox3: MATLAB tools that facilitate physically based stimulus rendering for vision research

RenderToolbox3 provides MATLAB utilities and prescribes a workflow that should be useful to researchers who want to employ graphics in the study of vision and perhaps in other endeavors as well. In particular, RenderToolbox3 facilitates rendering scene families in which various scene attributes and renderer behaviors are manipulated parametrically, enables spectral specification of object reflectance and illuminant spectra, enables the use of physically based material specifications, helps validate renderer output, and converts renderer output to physical units of radiance. This paper describes the design and functionality of the toolbox and discusses several examples that demonstrate its use. We have designed RenderToolbox3 to be portable across computer hardware and operating systems and to be free and open source (except for MATLAB itself). RenderToolbox3 is available at https://github.com/DavidBrainard/RenderToolbox3.

Reflectance, illumination, and appearance in color constancy

We studied color constancy using a pair of identical 3-D Color Mondrian displays. We viewed one 3-D Mondrian in nearly uniform illumination, and the other in directional, nonuniform illumination. We used the three dimensional structures to modulate the light falling on the painted surfaces. The 3-D structures in the displays were a matching set of wooden blocks. Across Mondrian displays, each corresponding facet had the same paint on its surface. We used only 6 chromatic, and 5 achromatic paints applied to 104 block facets. The 3-D blocks add shadows and multiple reflections not found in flat Mondrians. Both 3-D Mondrians were viewed simultaneously, side-by-side. We used two techniques to measure correlation of appearance with surface reflectance. First, observers made magnitude estimates of changes in the appearances of identical reflectances. Second, an author painted a watercolor of the 3-D Mondrians. The watercolor's reflectances quantified the changes in appearances. While constancy generalizations about illumination and reflectance hold for flat Mondrians, they do not for 3-D Mondrians. A constant paint does not exhibit perfect color constancy, but rather shows significant shifts in lightness, hue and chroma in response to the structure in the nonuniform illumination. Color appearance depends on the spatial information in both the illumination and the reflectances of objects. The spatial information of the quanta catch from the array of retinal receptors generates sensations that have variable correlation with surface reflectance. Models of appearance in humans need to calculate the departures from perfect constancy measured here. This article provides a dataset of measurements of color appearances for computational models of sensation.

Depth effect on lightness revisited: The role of articulation, proximity and fields of illumination

The coplanar ratio principle proposes that when the luminance range in an image is larger than the canonical reflectance range of 30:1, the lightness of a target surface depends on the luminance ratio between that target and its adjacent coplanar neighbor (Gilchrist, 1980). This conclusion is based on experiments in which changes in the perceived target depth produced large changes in its perceived lightness without significantly altering the observers' retinal image. Using the same paradigm, we explored how this depth effect on lightness depends on display complexity (articulation), proximity of the target to its highest coplanar luminance and spatial distribution of fields of illumination. Importantly, our experiments allowed us to test differing predictions made by the anchoring theory (Gilchrist et al., 1999), the coplanar ratio principle, as well as other models. We report three main findings, generally consistent with anchoring theory predictions: (1) Articulation can substantially increase the depth effect. (2) Target lightness depends not on the adjacent luminance but on the highest coplanar luminance, irrespective of its position relative to the target. (3) When a plane contains multiple fields of illumination, target lightness depends on the highest luminance in its field of illumination, not on the highest coplanar luminance.

The color constancy of three-dimensional objects

Human color constancy has been studied for over 100 years, and there is extensive experimental data for the case where a spatially diffuse light source illuminates a set of flat matte surfaces. In natural viewing, however, three-dimensional objects are viewed in three-dimensional scenes. Little is known about color constancy for three-dimensional objects. We used a forced-choice task to measure the achromatic chromaticity of matte disks, matte spheres, and glossy spheres. In all cases, the test stimuli were viewed in the context of stereoscopically viewed graphics simulations of three-dimensional scenes, and we varied the scene illuminant. We studied conditions both where all cues were consistent with the simulated illuminant change (consistent-cue conditions) and where local contrast was silenced as a cue (reduced-cue conditions). We computed constancy indices from the achromatic chromaticities. To first order, constancy was similar for the three test object types. There was, however, a reliable interaction between test object type and cue condition. In the consistent-cue conditions, constancy tended to be best for the matte disks, while in the reduced-cue conditions constancy was best for the spheres. The presence of this interaction presents an important challenge for theorists who seek to generalize models that account for constancy for flat tests to the more general case of three-dimensional objects.

Individual differences in simultaneous color constancy are related to working memory

Few studies have investigated the possible role of higher-level cognitive mechanisms in color constancy. Following up on previous work with successive color constancy [J. Exper. Psychol. Learn. Mem. Cogn. 37, 1014 (2011)], the current study examined the relation between simultaneous color constancy and working memory-the ability to maintain a desired representation while suppressing irrelevant information. Higher working memory was associated with poorer simultaneous color constancy of a chromatically complex stimulus. Ways in which the executive attention mechanism of working memory may account for this are discussed. This finding supports a role for higher-level cognitive mechanisms in color constancy and is the first to demonstrate a relation between simultaneous color constancy and a complex cognitive ability.

Working memory is related to perceptual processing: a case from color perception

We explored the relation between individual differences in working memory (WM) and color constancy, the phenomenon of color perception that allows us to perceive the color of an object as relatively stable under changes in illumination. Successive color constancy (measured by first viewing a colored surface under a particular illumination and later recalling it under a new illumination) was better for higher WM individuals than for lower WM individuals. Moreover, the magnitude of this WM difference depended on how much contextual information was available in the scene, which typically improves color constancy. By contrast, simple color memory, measured by viewing and recalling a colored surface under the same illumination, showed no significant relation to WM. This study reveals a relation between WM and a low-level perceptual process not previously thought to operate within the confines of attentional control, and it provides a first account of the individual differences in color constancy known about for decades.

Surface color perception and equivalent illumination models

Vision provides information about the properties and identity of objects. The ease with which we perceive object properties belies the difficulty of the underlying information-processing task. In the case of object color, retinal information about object reflectance is confounded with information about the illumination as well as about the object's shape and pose. There is no obvious rule that allows transformation of the retinal image to a color representation that depends primarily on object surface reflectance. Under many circumstances, however, object color appearance is remarkably stable across scenes in which the object is viewed. Here, we review a line of experiments and theory that aim to understand how the visual system stabilizes object color appearance. Our emphasis is on models derived from explicit analysis of the computational problem of estimating the physical properties of illuminants and surfaces from the retinal image, and experiments that test these models. We argue that this approach has considerable promise for allowing generalization from simplified laboratory experiments to richer scenes that more closely approximate natural viewing. We discuss the relation between the work we review and other theoretical approaches available in the literature.

Color and material perception: achievements and challenges

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

Color constancy

A quarter of a century ago, the first systematic behavioral experiments were performed to clarify the nature of color constancy-the effect whereby the perceived color of a surface remains constant despite changes in the spectrum of the illumination. At about the same time, new models of color constancy appeared, along with physiological data on cortical mechanisms and photographic colorimetric measurements of natural scenes. Since then, as this review shows, there have been many advances. The theoretical requirements for constancy have been better delineated and the range of experimental techniques has been greatly expanded; novel invariant properties of images and a variety of neural mechanisms have been identified; and increasing recognition has been given to the relevance of natural surfaces and scenes as laboratory stimuli. Even so, there remain many theoretical and experimental challenges, not least to develop an account of color constancy that goes beyond deterministic and relatively simple laboratory stimuli and instead deals with the intrinsically variable nature of surfaces and illuminations present in the natural world.

Effect of Scene Dimensionality on Colour Constancy with Real Three-Dimensional Scenes and Objects

The effect of scene dimensionality on colour constancy was tested with real scenes and objects. Observers viewed a three-dimensional (3-D) scene, or its two-dimensional (2-D) planar projection, through a large beam-splitter that projected the virtual image of a real test object (a cube or its 2-D projection) so that it appeared part of the scene. Test object and scene could be illuminated independently with high chromatic precision. In each trial, the illuminance of the scene changed abruptly from 25 000 K to 6700 K and the illuminant of the test object changed either consistently or inconsistently with it by a variable quantifiable amount. Observers had to decide whether the test object underwent a change in its materials. The extent of constancy obtained in the experiment was not influenced by scene dimensionality and varied significantly with the colour of the test object. These results suggest that color constancy in the conditions tested here may be determined by local spectral quantities.

Perception of Illuminant Colour Changes across Real Scenes

In a complex natural scene the colour and intensity of the illumination may vary considerably across the scene. Changes in intensity can easily be detected but the same does not seem to be true of colour changes. We investigated the extent to which chromatic changes of the illuminant are detected and the relation of detection performance with colour constancy and scene interpretation. The stimuli were complex real 3-D scenes rendered with spatial colour gradients of which the extremes had correlated colour temperatures within the range 25 000 K–3300 K. Observers' sensitivity to these spatial changes of the illuminant was found to be low and critically dependent on scene composition. Also, even in extreme conditions where colour constancy is known to fail, changes in the color of the illuminant across the scenes could not be perceived. These results suggest that insensitivity to spatial changes of the colour of the illuminant is a strong phenomenon and that it holds regardless of colour constancy.

Surface gloss and color perception of 3D objects

Two experiments explore the color perception of objects in complex scenes. The first experiment examines the color perception of objects across variation in surface gloss. Observers adjusted the color appearance of a matte sphere to match that of a test sphere. Across conditions we varied the body color and glossiness of the test sphere. The data indicate that observers do not simply match the average light reflected from the test. Indeed, the visual system compensates for the physical effect of varying the gloss, so that appearance is stabilized relative to what is predicted by the spatial average. The second experiment examines how people perceive color across locations on an object. We replaced the test sphere with a soccer ball that had one of its hexagonal faces colored. Observers were asked to adjust the match sphere have the same color appearance as this test patch. The test patch could be located at either an upper or lower location on the soccer ball. In addition, we varied the surface gloss of the entire soccer ball (including the test patch). The data show that there is an effect of test patch location on observers' color matching, but this effect is small compared to the physical change in the average light reflected from the test patch across the two locations. In addition, the effect of glossy highlights on the color appearance of the test patch was consistent with the results from Experiment 1.

Color constancy: phenomenal or projective?

Naive observers viewed a sequence of colored Mondrian patterns, simulated on a color monitor. Each pattern was presented twice in succession, first under one daylight illuminant with a correlated color temperature of either 16,000 or 4000 K and then under the other, to test for color constancy. The observers compared the central square of the pattern across illuminants, either rating it for sameness of material appearance or sameness of hue and saturation or judging an objective property-that is, whether its change of color originated from a change in material or only from a change in illumination. Average color constancy indices were high for material appearance ratings and binary judgments of origin and low for hue-saturation ratings. Individuals' performance varied, but judgments of material and of hue and saturation remained demarcated. Observers seem able to separate phenomenal percepts from their ontological projections of mental appearance onto physical phenomena; thus, even when a chromatic change alters perceived hue and saturation, observers can reliably infer the cause, the constancy of the underlying surface spectral reflectance.

Creating physically accurate visual stimuli for free: spectral rendering with RADIANCE

Visual psychophysicists, who study object, color, and light perception, have a demand for software that produces complex but, at the same time, physically accurate stimuli for their experiments. The number of computer graphic packages that simulate the physical interaction of light and surfaces is limited, and mostly they require the purchase of a license. RADIANCE (Ward, 1994), however, is freely available and popular in the visual perception community, making it a prime candidate. We have shown previously that RADIANCE's simulation accuracy is greatly improved when color is coded by spectra, rather than by the originally envisaged RGB triplets (Ruppertsberg & Bloj, 2006). Here, we present a method for spectral rendering with RADIANCE to generate hyperspectral images that can be converted to XYZ images (CIE 1931 system) and then to machine-dependent RGB images. Generating XYZ stimuli has the added advantage of making stimulus images independent of display devices and, thereby, facilitating the process of reproducing results across different labs. Materials associated with this article may be downloaded from www.psychonomic.org.

Coding of images of materials by macaque inferior temporal cortical neurons

Objects vary not only in their shape but also in the material from which they are made. Knowledge of the material properties can contribute to object recognition as well as indicate properties of the object (e.g. ripeness of a fruit). We examined the coding of images of materials by single neurons of the macaque inferior temporal (IT) cortex, an area known to support object recognition and categorization. Stimuli were images of 12 real materials that were illuminated from three different directions. The material textures appeared within five different outline shapes. The majority of responsive IT neurons responded selectively to the material textures, and this selectivity was largely independent of their shape selectivity. The responses of the large majority of neurons were strongly affected by illumination direction. Despite the generally weak illumination-direction invariance of the responses, Support Vector Machines that used the neural responses as input were able to classify the materials across illumination direction better than by chance. A comparison between the responses to the original images and those to images with a random spectral phase, but matched power spectrum, indicated that the material texture selectivity did not depend merely on differences in the power spectrum but required phase information.

Testing limits on matte surface color perception in three-dimensional scenes with complex light fields

We investigated limits on the human visual system's ability to discount directional variation in complex lights field when estimating Lambertian surface color. Directional variation in the light field was represented in the frequency domain using spherical harmonics. The bidirectional reflectance distribution function of a Lambertian surface acts as a low-pass filter on directional variation in the light field. Consequently, the visual system needs to discount only the low-pass component of the incident light corresponding to the first nine terms of a spherical harmonics expansion [Basri, R., Jacobs, D. (2001). Lambertian reflectance and linear subspaces. In: International Conference on Computer Vision II, pp. 383-390; Ramamoorthi, R., Hanrahan, P., (2001). An efficient representation for irradiance environment maps. SIGGRAPH 01. New York: ACM Press, pp. 497-500] to accurately estimate surface color. We test experimentally whether the visual system discounts directional variation in the light field up to this physical limit. Our results are consistent with the claim that the visual system can compensate for all of the complexity in the light field that affects the appearance of Lambertian surfaces.

Bayesian model of human color constancy

Vision is difficult because images are ambiguous about the structure of the world. For object color, the ambiguity arises because the same object reflects a different spectrum to the eye under different illuminations. Human vision typically does a good job of resolving this ambiguity-an ability known as color constancy. The past 20 years have seen an explosion of work on color constancy, with advances in both experimental methods and computational algorithms. Here, we connect these two lines of research by developing a quantitative model of human color constancy. The model includes an explicit link between psychophysical data and illuminant estimates obtained via a Bayesian algorithm. The model is fit to the data through a parameterization of the prior distribution of illuminant spectral properties. The fit to the data is good, and the derived prior provides a succinct description of human performance.

Surface color perception in three-dimensional scenes

Researchers studying surface color perception have typically used stimuli that consist of a small number of matte patches (real or simulated) embedded in a plane perpendicular to the line of sight (a "Mondrian," Land & McCann, 1971). Reliable estimation of the color of a matte surface is a difficult if not impossible computational problem in such limited scenes (Maloney, 1999). In more realistic, three-dimensional scenes the difficulty of the problem increases, in part, because the effective illumination incident on the surface (the light field) now depends on surface orientation and location. We review recent work in multiple laboratories that examines (1) the degree to which the human visual system discounts the light field in judging matte surface lightness and color and (2) what illuminant cues the visual system uses in estimating the flow of light in a scene.

Color constancy in natural scenes with and without an explicit illuminant cue

Observers can generally make reliable judgments of surface color in natural scenes despite changes in an illuminant that is out of view. This ability has sometimes been attributed to observers' estimating the spectral properties of the illuminant in order to compensate for its effects. To test this hypothesis, two surface-color-matching experiments were performed with images of natural scenes obtained from high-resolution hyperspectral images. In the first experiment, the sky illuminating the scene was directly visible to the observer, and its color was manipulated. In the second experiment, a large gray sphere was introduced into the scene so that its illumination by the sun and sky was also directly visible to the observer, and the color of that illumination was manipulated. Although the degree of color constancy varied across this and other variations of the images, there was no reliable effect of illuminant color. Even when the sky was eliminated from view, color constancy did not worsen. Judging surface color in natural scenes seems to be independent of an explicit illuminant cue.

How direction of illumination affects visually perceived surface roughness

We examined visual estimation of surface roughness using random, computer-generated, three-dimensional (3D) surfaces rendered under a mixture of diffuse lighting and a punctate source. The angle between the tangent to the plane containing the surface texture and the direction to the punctate source was varied from 50 to 70 deg across lighting conditions. Observers were presented with pairs of surfaces under different lighting conditions and indicated which 3D surface appeared rougher. Surfaces were viewed either in isolation or in scenes with added objects whose shading, cast shadows, and specular highlights provided information about the spatial distribution of illumination. All observers perceived surfaces to be markedly rougher with decreasing illuminant angle. Performance in scenes with added objects was no closer to constant than that in scenes without added objects. We identified four novel cues that are valid cues to roughness under any single lighting condition but that are not invariant under changes in lighting condition. We modeled observers' deviations from roughness constancy as a weighted linear combination of these "pseudocues" and found that they account for a substantial amount of observers' systematic deviations from roughness constancy with changes in lighting condition.

Rendering complex scenes for psychophysics using RADIANCE: how accurate can you get?

Rendering packages are used by visual psychophysicists to produce complex stimuli for their experiments, tacitly assuming that the simulation results accurately reflect the light-surface interactions of a real scene. RADIANCE is a physically based, freely available, and commonly used rendering software. We validated the calculation accuracy of this package by comparing simulation results with measurements from real scenes. RADIANCE recovers color gradients well but the results are shifted in color space. Currently, there is no better simulation alternative for achieving physical accuracy than by combining a spectral rendering method with RADIANCE.

Lightness identification of patterned three-dimensional, real objects

Conventional studies of lightness constancy have almost exclusively used flat plain stimuli and have shown that lightness matches across illuminants cannot be explained by physical matches of reflectance or luminance. The perceptual qualities that underlie lightness judgments still remain largely unknown. Real objects are often 3-D and patterned, giving additional cues for identification. We examine the perceptual strategies that underlie material identification of real objects. Stimuli were randomly crumpled papers printed with achromatic patterns with precisely calibrated mean reflectance and reflectance contrast, placed in backgrounds under varying levels of illumination. Observers were asked to identify objects based on physical reflectance differences. Reflectance identification functions were simulated by simple models that perform object identification based on dissimilarities in perceived brightness (luminance dissimilarity modified by light adaptation) or perceived contrast (contrast dissimilarity modified by mean luminance). The reflectance identification results were also recreated in two control experiments, using identical stimuli conditions, where choices were based explicitly on dissimilarities in perceived brightness or contrast. Rather than a reverse optics model of lightness perception where observers first estimate illuminant intensity and then extract relative lightness by discounting the illuminant, this study supports the use of simple percepts such as brightness and contrast.

Color Perception of 3D Objects: Constancy with Respect To Variation of Surface Gloss

What determines the color appearance of real objects viewed under natural conditions? The light reflected from different locations on a single object can vary enormously. This variation is enhanced when the material properties of the object are changed from matte to glossy. Yet humans have no trouble assigning a color name to most things. We studied how people perceive the color of spheres in complex scenes. Observers viewed graphics simulations of a three-dimensional scene containing two spheres, test and match. The observer's task was to adjust the match sphere until its color appearance was the same as that of the test sphere. The match sphere was always matte, and observers varied its color by changing the simulated spectral reflectance function. The surface gloss of the test spheres was varied across conditions. The data show that for fixed test sphere body reflectance, color appearance depends on surface gloss. This effect is small, however, compared to the variation that would be expected if observers simply matched the average of the light reflected from the test.

Sensory, computational and cognitive components of human colour constancy

When the illumination on a scene changes, so do the visual signals elicited by that scene. In spite of these changes, the objects within a scene tend to remain constant in their apparent colour. We start this review by discussing the psychophysical procedures that have been used to quantify colour constancy. The transformation imposed on the visual signals by a change in illumination dictates what the visual system must 'undo' to achieve constancy. The problem is mathematically underdetermined, and can be solved only by exploiting regularities of the visual world. The last decade has seen a substantial increase in our knowledge of such regularities as technical advances have made it possible to make empirical measurements of large numbers of environmental scenes and illuminants. This review provides a taxonomy of models of human colour constancy based first on the assumptions they make about how the inverse transformation might be simplified, and second, on how the parameters of the inverse transformation might be set by elements of a complex scene. Candidate algorithms for human colour constancy are represented graphically and pictorially, and the availability and utility of an accurate estimate of the illuminant is discussed. Throughout this review, we consider both the information that is, in principle, available and empirical assessments of what information the visual system actually uses. In the final section we discuss where in our visual systems these computations might be implemented.

Does colour constancy exist?

For a stable visual world, the colours of objects should appear the same under different lights. This property of colour constancy has been assumed to be fundamental to vision, and many experimental attempts have been made to quantify it. I contend here, however, that the usual methods of measurement are either too coarse or concentrate not on colour constancy itself, but on other, complementary aspects of scene perception. Whether colour constancy exists other than in nominal terms remains unclear.

Colour constancy under simultaneous changes in surface position and illuminant

Two kinds of constancy underlie the everyday perception of surface colour: constancy under changes in illuminant and constancy under changes in surface position. Classically, these two constancies seem to place conflicting demands on the visual system: to both take into account the region surrounding a surface and also discount it. It is shown here, however, that the ability of observers to make surface-colour matches across simultaneous changes in test-surface position and illuminant in computer-generated 'Mondrian' patterns is almost as good as across changes in illuminant alone. Performance was no poorer when the surfaces surrounding the test surface were permuted, or when information from a potential comparison surface, the one with the highest luminance, was suppressed. Computer simulations of cone-photoreceptor activity showed that a reliable cue for making surface-colour matches in all experimental conditions was provided by the ratios of cone excitations between the test surfaces and a spatial average over the whole pattern.