Parallel detection of violations of color constancy

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

Colour constancy failures expected in colourful environments

Colour constancy refers to the constant perceived or apparent colour of a surface despite changes in illumination spectrum. Laboratory measurements have often found it imperfect. The aim here was to estimate the frequency of constancy failures in natural outdoor environments and relate them to colorimetric surface properties. A computational analysis was performed with 50 hyperspectral reflectance images of outdoor scenes undergoing simulated daylight changes. For a chromatically adapted observer, estimated colour appearance changed noticeably for at least 5% of the surface area in 60% of scenes, and at least 10% of the surface area in 44% of scenes. Somewhat higher frequencies were found for estimated changes in perceived colour relations represented by spatial ratios of cone-photoreceptor excitations. These estimated changes correlated with surface chroma and saturation. Outdoors, the colour constancy of some individual surfaces seems likely to fail, particularly if those surfaces are colourful.

Disentangling simultaneous changes of surface and illumination

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Motion of glossy objects does not promote separation of lighting and surface colour

The surface properties of an object, such as texture, glossiness or colour, provide important cues to its identity. However, the actual visual stimulus received by the eye is determined by both the properties of the object and the illumination. We tested whether operational colour constancy for glossy objects (the ability to distinguish changes in spectral reflectance of the object, from changes in the spectrum of the illumination) was affected by rotational motion of either the object or the light source. The different chromatic and geometric properties of the specular and diffuse reflections provide the basis for this discrimination, and we systematically varied specularity to control the available information. Observers viewed animations of isolated objects undergoing either lighting or surface-based spectral transformations accompanied by motion. By varying the axis of rotation, and surface patterning or geometry, we manipulated: (i) motion-related information about the scene, (ii) relative motion between the surface patterning and the specular reflection of the lighting, and (iii) image disruption caused by this motion. Despite large individual differences in performance with static stimuli, motion manipulations neither improved nor degraded performance. As motion significantly disrupts frame-by-frame low-level image statistics, we infer that operational constancy depends on a high-level scene interpretation, which is maintained in all conditions.

Low levels of specularity support operational color constancy, particularly when surface and illumination geometry can be inferred

We tested whether surface specularity alone supports operational color constancy-the ability to discriminate changes in illumination or reflectance. Observers viewed short animations of illuminant or reflectance changes in rendered scenes containing a single spherical surface and were asked to classify the change. Performance improved with increasing specularity, as predicted from regularities in chromatic statistics. Peak performance was impaired by spatial rearrangements of image pixels that disrupted the perception of illuminated surfaces but was maintained with increased surface complexity. The characteristic chromatic transformations that are available with nonzero specularity are useful for operational color constancy, particularly if accompanied by appropriate perceptual organization.

Time-lapse ratios of cone excitations in natural scenes

The illumination in natural environments varies through the day. Stable inferences about surface color might be supported by spatial ratios of cone excitations from the reflected light, but their invariance has been quantified only for global changes in illuminant spectrum. The aim here was to test their invariance under natural changes in both illumination spectrum and geometry, especially in the distribution of shadows. Time-lapse hyperspectral radiance images were acquired from five outdoor vegetated and nonvegetated scenes. From each scene, 10,000 pairs of points were sampled randomly and ratios measured across time. Mean relative deviations in ratios were generally large, but when sampling was limited to short distances or moderate time intervals, they fell below the level for detecting violations in ratio invariance. When illumination changes with uneven geometry were excluded, they fell further, to levels obtained with global changes in illuminant spectrum alone. Within sampling constraints, ratios of cone excitations, and also of opponent-color combinations, provide an approximately invariant signal for stable surface-color inferences, despite spectral and geometric variations in scene illumination.

Lightness constancy and its link with cone contrast

The link between chromatic constancy (compensation for hue and saturation shifts) and lightness constancy (compensation for a change in surface reflectance) was tested theoretically by computing cone contrasts and by asymmetric matching experiments. The effect of a thin achromatic line (a frame) around the test sample was tested empirically. When the samples were outlined by the frame, lightness constancy was increased and chromatic constancy reduced (p<0.001). Changes in luminance are more likely to be compensated when the luminance contrast edge around the test stimulus is disturbed as with the addition of an achromatic frame.

Influence of local scene color on fixation position in visual search

Where observers concentrate their gaze during visual search depends on several factors. The aim here was to determine how much of the variance in observers' fixations in natural scenes can be explained by local scene color and how that variance is related to viewing bias. Fixation data were taken from an experiment in which observers searched images of 20 natural rural and urban scenes for a small target. The proportion R2 of the variance explained in a regression on local color properties (lightness and the red-green and yellow-blue chromatic components) ranged from 1% to 85%, depending mainly on how well those properties were consistent with observers' viewing bias. When viewing bias was included in the regression, values of R2 increased, ranging from 62% to 96%. By comparison, local lightness and local lightness contrast, edge density, and entropy each explained less variance than local color properties. Local scene color may have a much stronger influence on gaze position than is generally recognized, capturing significant aspects of scene structure on target search behavior.

Context-dependent judgments of color that might allow color constancy in scenes with multiple regions of illumination

For a color-constant observer, a change in the spectral composition of the illumination is accompanied by a corresponding change in the chromaticity associated with an achromatic percept. However, maintaining color constancy for different regions of illumination within a scene implies the maintenance of multiple perceptual references. We investigated the features of a scene that enable the maintenance of separate perceptual references for two displaced but overlapping chromaticity distributions. The time-averaged, retinotopically localized stimulus was the primary determinant of color appearance judgments. However, spatial separation of test samples additionally served as a symbolic cue that allowed observers to maintain two separate perceptual references.

Slow updating of the achromatic point after a change in illumination

For a color constant observer, the color appearance of a surface is independent of the spectral composition of the light illuminating it. We ask how rapidly color appearance judgments are updated following a change in illumination. We obtained repeated binary color classifications for a set of stimuli defined by their reflectance functions and rendered under either sunlight or skylight. We used these classifications to derive boundaries in color space that identify the observer's achromatic point. In steady-state conditions of illumination, the achromatic point lay close to the illuminant chromaticity. In our experiment, the illuminant changed abruptly every 21 s (at the onset of every 10th trial), allowing us to track changes in the achromatic point that were caused by the cycle of illuminant changes. In one condition, the test reflectance was embedded in a spatial pattern of reflectance samples under consistent illumination. The achromatic point migrated across color space between the chromaticities of the steady-state achromatic points. This update took several trials rather than being immediate. To identify the factors that governed perceptual updating of appearance judgments, we used two further conditions, one in which the test reflectance was presented in isolation and one in which the surrounding reflectances were rendered under an inconsistent and unchanging illumination. Achromatic settings were not well predicted by the information available from scenes at a single time point. Instead, the achromatic points showed a strong dependence on the history of chromatic samples. The strength of this dependence differed between observers and was modulated by the spatial context.

Hue manifold

It is generally accepted that hues can be arranged so as to make a circle. The circular representation of hue has been supported by multidimensional scaling, which allows for the representation of a set of colored papers as a configuration in a Euclidean space where the distances between the papers correspond to the perceptual dissimilarities between them. In particular, when papers of various hues are evenly illuminated, they are arranged in a one-dimensional circular configuration. However, under variegated illumination we show that the same papers in fact make a two-dimensional configuration that resembles a torus.

Color constancy

A quarter of a century ago, the first systematic behavioral experiments were performed to clarify the nature of color constancy-the effect whereby the perceived color of a surface remains constant despite changes in the spectrum of the illumination. At about the same time, new models of color constancy appeared, along with physiological data on cortical mechanisms and photographic colorimetric measurements of natural scenes. Since then, as this review shows, there have been many advances. The theoretical requirements for constancy have been better delineated and the range of experimental techniques has been greatly expanded; novel invariant properties of images and a variety of neural mechanisms have been identified; and increasing recognition has been given to the relevance of natural surfaces and scenes as laboratory stimuli. Even so, there remain many theoretical and experimental challenges, not least to develop an account of color constancy that goes beyond deterministic and relatively simple laboratory stimuli and instead deals with the intrinsically variable nature of surfaces and illuminations present in the natural world.

Effect of Scene Dimensionality on Colour Constancy with Real Three-Dimensional Scenes and Objects

The effect of scene dimensionality on colour constancy was tested with real scenes and objects. Observers viewed a three-dimensional (3-D) scene, or its two-dimensional (2-D) planar projection, through a large beam-splitter that projected the virtual image of a real test object (a cube or its 2-D projection) so that it appeared part of the scene. Test object and scene could be illuminated independently with high chromatic precision. In each trial, the illuminance of the scene changed abruptly from 25 000 K to 6700 K and the illuminant of the test object changed either consistently or inconsistently with it by a variable quantifiable amount. Observers had to decide whether the test object underwent a change in its materials. The extent of constancy obtained in the experiment was not influenced by scene dimensionality and varied significantly with the colour of the test object. These results suggest that color constancy in the conditions tested here may be determined by local spectral quantities.

Color constancy: phenomenal or projective?

Naive observers viewed a sequence of colored Mondrian patterns, simulated on a color monitor. Each pattern was presented twice in succession, first under one daylight illuminant with a correlated color temperature of either 16,000 or 4000 K and then under the other, to test for color constancy. The observers compared the central square of the pattern across illuminants, either rating it for sameness of material appearance or sameness of hue and saturation or judging an objective property-that is, whether its change of color originated from a change in material or only from a change in illumination. Average color constancy indices were high for material appearance ratings and binary judgments of origin and low for hue-saturation ratings. Individuals' performance varied, but judgments of material and of hue and saturation remained demarcated. Observers seem able to separate phenomenal percepts from their ontological projections of mental appearance onto physical phenomena; thus, even when a chromatic change alters perceived hue and saturation, observers can reliably infer the cause, the constancy of the underlying surface spectral reflectance.

Representation or Context as a Cognitive Strategy in Colour Constancy?

If an identification task with colour constancy as its objective is carried out under drastically changing illumination, do people rely mainly on colour information or do they rely on other sources of information? This question suggested two hypotheses for testing: (i) context hypothesis: people rely mainly on colour information (spectral reflectance or illumination chromaticity) to achieve colour constancy; (ii) representation hypothesis: people rely mainly on all other clues associated with colour to achieve colour constancy, including form information (any shape elements) and space information (spatial coordinates or spatial correlation). Experiment 1 showed that form information was readily associated with colour information to produce implicit representation. This gave the best colour-constancy performance (95.72%) and the fastest processing speed, so it probably used a top–down process. However, it was also prone to error owing to assumptions. Space information was not readily associated with colour information so colour-constancy performance was halved (48.73%) and processing time doubled. When the subject was deprived of both information sources and only given colour information, this resulted in the longest reaction times and the worst colour-constancy performance (41.38%). These results clearly support the representation hypothesis rather than the context hypothesis. When all three clues were available at the same time, the order of preference was: symbol, location, then colour. Experiment 2 showed that when form information was the main clue, colour-constancy performance was conceptually driven and processed more quickly; this supports the representation hypothesis. However, when form information was not used, colour constancy was data-driven, processed more slowly, and achieved an inferior identification rate overall; this supports the context hypothesis.

Color constancy in natural scenes explained by global image statistics

To what extent do observers' judgments of surface color with natural scenes depend on global image statistics? To address this question, a psychophysical experiment was performed in which images of natural scenes under two successive daylights were presented on a computer-controlled high-resolution color monitor. Observers reported whether there was a change in reflectance of a test surface in the scene. The scenes were obtained with a hyperspectral imaging system and included variously trees, shrubs, grasses, ferns, flowers, rocks, and buildings. Discrimination performance, quantified on a scale of 0 to 1 with a color-constancy index, varied from 0.69 to 0.97 over 21 scenes and two illuminant changes, from a correlated color temperature of 25,000 K to 6700 K and from 4000 K to 6700 K. The best account of these effects was provided by receptor-based rather than colorimetric properties of the images. Thus, in a linear regression, 43% of the variance in constancy index was explained by the log of the mean relative deviation in spatial cone-excitation ratios evaluated globally across the two images of a scene. A further 20% was explained by including the mean chroma of the first image and its difference from that of the second image and a further 7% by the mean difference in hue. Together, all four global color properties accounted for 70% of the variance and provided a good fit to the effects of scene and of illuminant change on color constancy, and, additionally, of changing test-surface position. By contrast, a spatial-frequency analysis of the images showed that the gradient of the luminance amplitude spectrum accounted for only 5% of the variance.

Anomalous trichromats' judgments of surface color in natural scenes under different daylights

Deuteranomalous trichromacy, which affects medium-wavelength-sensitive cones, is more common than protanomalous trichromacy, which affects long-wavelength-sensitive cones. The aim of the present work was to test the extent to which these two kinds of anomalous trichromacy affect surface-color judgments in the natural world. Simulations of 18 natural scenes under different daylight illuminants were presented on a high-resolution color monitor to 7 deuteranomalous, 7 protanomalous, and 12 normal trichromatic observers, who had to discriminate between reflectance and illuminant changes in the images. Observers' ability to judge surface color was quantified by a standard color-constancy index. Deuteranomalous trichromats performed as well as normal trichromats, but protanomalous trichromats performed more poorly than both. The results are considered in relation to the spectral coverage of cones, rod intrusion, and the characterization of anomalous trichromacy by the Rayleigh match.

Color constancy in natural scenes with and without an explicit illuminant cue

Observers can generally make reliable judgments of surface color in natural scenes despite changes in an illuminant that is out of view. This ability has sometimes been attributed to observers' estimating the spectral properties of the illuminant in order to compensate for its effects. To test this hypothesis, two surface-color-matching experiments were performed with images of natural scenes obtained from high-resolution hyperspectral images. In the first experiment, the sky illuminating the scene was directly visible to the observer, and its color was manipulated. In the second experiment, a large gray sphere was introduced into the scene so that its illumination by the sun and sky was also directly visible to the observer, and the color of that illumination was manipulated. Although the degree of color constancy varied across this and other variations of the images, there was no reliable effect of illuminant color. Even when the sky was eliminated from view, color constancy did not worsen. Judging surface color in natural scenes seems to be independent of an explicit illuminant cue.

Sensory, computational and cognitive components of human colour constancy

When the illumination on a scene changes, so do the visual signals elicited by that scene. In spite of these changes, the objects within a scene tend to remain constant in their apparent colour. We start this review by discussing the psychophysical procedures that have been used to quantify colour constancy. The transformation imposed on the visual signals by a change in illumination dictates what the visual system must 'undo' to achieve constancy. The problem is mathematically underdetermined, and can be solved only by exploiting regularities of the visual world. The last decade has seen a substantial increase in our knowledge of such regularities as technical advances have made it possible to make empirical measurements of large numbers of environmental scenes and illuminants. This review provides a taxonomy of models of human colour constancy based first on the assumptions they make about how the inverse transformation might be simplified, and second, on how the parameters of the inverse transformation might be set by elements of a complex scene. Candidate algorithms for human colour constancy are represented graphically and pictorially, and the availability and utility of an accurate estimate of the illuminant is discussed. Throughout this review, we consider both the information that is, in principle, available and empirical assessments of what information the visual system actually uses. In the final section we discuss where in our visual systems these computations might be implemented.

Does colour constancy exist?

For a stable visual world, the colours of objects should appear the same under different lights. This property of colour constancy has been assumed to be fundamental to vision, and many experimental attempts have been made to quantify it. I contend here, however, that the usual methods of measurement are either too coarse or concentrate not on colour constancy itself, but on other, complementary aspects of scene perception. Whether colour constancy exists other than in nominal terms remains unclear.

Minimum-variance cone-excitation ratios and the limits of relational color constancy

Relational color constancy refers to the constancy of the perceived relations between the colors of surfaces of a scene under changes in the spectral composition of the illuminant. Spatial ratios of cone excitations provide a natural physical basis for this constancy, as, on average, they are almost invariant under illuminant changes for large collections of natural surfaces and illuminants. The aim of the present work was to determine, computationally, for specific surfaces and illuminants, the constancy limits obtained by the application of a minimum-variance principle to cone-excitation ratios and to investigate its validity in predicting observers' surface-color judgments. Cone excitations and their changes due to variations in the color of the illuminant were estimated for colored surfaces in simulated two-dimensional scenes of colored papers and real three-dimensional scenes of solid colored objects. For various test surfaces, scenes, and illuminants, the estimated levels of relational color constancy mediated by cone-excitation ratios varied significantly with the test surface and only with certain desaturated surfaces corresponded to ideal matches. Observers' experimental matches were compared with predictions expressed in CIE 1976 (u′,v′) space and were found to be generally consistent with minimum-variance predictions.

Information limits on neural identification of colored surfaces in natural scenes

If surfaces in a scene are to be distinguished by their color, their neural representation at some level should ideally vary little with the color of the illumination. Four possible neural codes were considered: von-Kries-scaled cone responses from single points in a scene, spatial ratios of cone responses produced by light reflected from pairs of points, and these quantities obtained with sharpened (opponent-cone) responses. The effectiveness of these codes in identifying surfaces was quantified by information-theoretic measures. Data were drawn from a sample of 25 rural and urban scenes imaged with a hyperspectral camera, which provided estimates of surface reflectance at 10-nm intervals at each of 1344 × 1024 pixels for each scene. In computer simulations, scenes were illuminated separately by daylights of correlated color temperatures 4000 K, 6500 K, and 25,000 K. Points were sampled randomly in each scene and identified according to each of the codes. It was found that the maximum information preserved under illuminant changes varied with the code, but for a particular code it was remarkably stable across the different scenes. The standard deviation over the 25 scenes was, on average, approximately 1 bit, suggesting that the neural coding of surface color can be optimized independent of location for any particular range of illuminants.

Colour constancy under simultaneous changes in surface position and illuminant

Two kinds of constancy underlie the everyday perception of surface colour: constancy under changes in illuminant and constancy under changes in surface position. Classically, these two constancies seem to place conflicting demands on the visual system: to both take into account the region surrounding a surface and also discount it. It is shown here, however, that the ability of observers to make surface-colour matches across simultaneous changes in test-surface position and illuminant in computer-generated 'Mondrian' patterns is almost as good as across changes in illuminant alone. Performance was no poorer when the surfaces surrounding the test surface were permuted, or when information from a potential comparison surface, the one with the highest luminance, was suppressed. Computer simulations of cone-photoreceptor activity showed that a reliable cue for making surface-colour matches in all experimental conditions was provided by the ratios of cone excitations between the test surfaces and a spatial average over the whole pattern.

Protanopic observers show nearly normal color constancy with natural reflectance spectra

The ability of color-deficient observers to discriminate between illuminant changes and surface-reflectance changes in a scene was tested with natural and Munsell reflectance spectra. To avoid the confounding effects of spatial structure, stimuli were simulations of Mondrian-like colored patterns, presented on a computer-controlled color monitor. Protanopes performed less well than normal trichromats, regardless of the type of reflectance spectra, but they were least disadvantaged with patterns comprising reflectance spectra drawn from urban and rural scenes, more characteristic of the natural environment.

Chromatic light adaptation measured using functional magnetic resonance imaging

Sensitivity changes, beginning at the first stages of visual transduction, permit neurons with modest dynamic range to respond to contrast variations across an enormous range of mean illumination. We have used functional magnetic resonance imaging (fMRI) to investigate how these sensitivity changes are controlled within the visual pathways. We measured responses in human visual area V1 to a constant-amplitude, contrast-reversing probe presented on a range of mean backgrounds. We found that signals from probes initiated in the L and M cones were affected by backgrounds that changed the mean absorption rates in the L and M cones, but not by background changes seen only by the S cones. Similarly, signals from S cone-initiated probes were altered by background changes in the S cones, but not by background changes in the L and M cones. Performance in psychophysical tests under similar conditions closely mirrored the changes in V1 fMRI signals. We compare our data with simulations of the visual pathway from photon catch rates to cortical blood-oxygen level-dependent signals and show that the quantitative fMRI signals are consistent with a simple model of mean-field adaptation based on Naka-Rushton (Naka and Rushton, 1966) adaptation mechanisms within cone photoreceptor classes.

Chromatic light adaptation measured using functional magnetic resonance imaging

Sensitivity changes, beginning at the first stages of visual transduction, permit neurons with modest dynamic range to respond to contrast variations across an enormous range of mean illumination. We have used functional magnetic resonance imaging (fMRI) to investigate how these sensitivity changes are controlled within the visual pathways. We measured responses in human visual area V1 to a constant-amplitude, contrast-reversing probe presented on a range of mean backgrounds. We found that signals from probes initiated in the L and M cones were affected by backgrounds that changed the mean absorption rates in the L and M cones, but not by background changes seen only by the S cones. Similarly, signals from S cone-initiated probes were altered by background changes in the S cones, but not by background changes in the L and M cones. Performance in psychophysical tests under similar conditions closely mirrored the changes in V1 fMRI signals. We compare our data with simulations of the visual pathway from photon catch rates to cortical blood-oxygen level-dependent signals and show that the quantitative fMRI signals are consistent with a simple model of mean-field adaptation based on Naka-Rushton (Naka and Rushton, 1966) adaptation mechanisms within cone photoreceptor classes.

Statistics of spatial cone-excitation ratios in natural scenes

For some sets of surfaces, the spatial ratios of cone-photoreceptor excitations produced by light reflected from pairs of surfaces are almost invariant under illuminant changes. These sets include large populations of spectral reflectances, some of which represent individual natural surfaces but not their relative abundances in nature. The aim of this study was to determine whether spatial cone-excitation ratios are preserved under illuminant changes within the natural visual environment. A fast hyperspectral imaging system was used to obtain populations of 640,000 reflectance spectra from each of 30 natural scenes. The statistics of spatial cone-excitation ratios for randomly selected pairs of points in these scenes were determined for two extreme daylights. Almost-invariant ratios were common, suggesting that they represent a reliable property of the natural visual environment and a suitable foundation for visual color constancy.