Colour constancy from temporal cues: better matches with less variability under fast illuminant changes

Color constancy in real-world settings

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

Temporal dynamics of daylight perception: Detection thresholds

Temporal changes in illumination are ubiquitous; natural light, for example, varies in color temperature and irradiance throughout the day. Yet little is known about human sensitivity to temporal changes in illumination spectra. Here, we aimed to determine the minimum detectable velocity of chromaticity change of daylight metamers in an immersive environment. The main stimulus was a continuous, monotonic change in global illumination chromaticity along the daylight locus in warmer (toward lower correlated color temperatures [CCTs]) or cooler directions, away from an adapting base light (CCT: 13,000 K, 6500 K, 4160 K, or 2000 K). All lights were generated by spectrally tunable overhead lamps as smoothest-possible metamers of the desired chromaticities. Mean detection thresholds (for 22 participants) for a fixed duration of 10 seconds ranged from 15 to 2 CIELUV ΔE units, depending significantly on base light CCT and with a significant interaction between CCT and direction of change. Cool changes become less noticeable for progressively warmer base lights and vice versa. For the two extreme base lights, sensitivity to changes toward neutral is significantly lower than for the opposite direction. The results suggest a “neutral bias” in illumination change discriminability, and that typical temporal changes in daylight chromaticity are likely to be below threshold detectability, at least where there are no concomitant overall illuminance changes. These factors may contribute to perceptual stability of natural scenes and color constancy.

Transparent layer constancy is improved by motion, stereo disparity, highly regular background pattern, and successive presentation

The visual system uses figural and colorimetric regularities in the retinal image to recognize optical filters and to discern the properties of the transparent overlay from properties of the background. Previous work suggests that the perceived color and transmittance of the transparent layer vary less under illumination changes than it would be expected from corresponding changes in the input. Here, we tested how the degree of this approximate transparent layer constancy (TLC) depends on factors that presumably facilitate the decomposition into a filter and a background layer. Using an asymmetric filter matching task, we found that motion, stereo disparity, and a highly regular background pattern each contribute to the vividness of the transparency impression and the degree of TLC. Combining these cues led to a cumulative increase in TLC, suggesting a "strong fusion" cue integration process. We also tested objects with invalid figural conditions for transparency (T-junctions). The tendency to perceive these objects as opaque and to establish a proximal match increased the more conspicuous the violation of this figural condition was. Furthermore, we investigated the gain in TLC due to alternating presentation. Alternating presentation enhanced TLC and color constancy to a comparable degree, and our results suggest that adaptation contributes to this effect.

Color and brightness constancies depend reciprocally on saturation

Color and brightness constancies may not be independent, since increasing a sample saturation should decrease the demand on the color constancy and increase that on the brightness constancy [Color Res. Appl.43, 630 (2018)CREADU0361-231710.1002/col.22227]. We tested this claim using color and brightness constancy data from Foster et al. [Vis. Res.41, 285 (2001)VISRAM0042-698910.1016/S0042-6989(00)00239-X], whose observers made side-by-side and successive comparisons of central patches ("asymmetric matches") presented in pairs of identical Mondrian displays with simulated illuminants of 25000 K and 6700 K daylights. Saturations (CIE "chroma") of the central patches varied from 0.007 to 0.092. For most observers (as in a toy Gaussian model of the surface reflectance spectra), increasing saturation reduced color constancy and increased brightness constancy.

Effect of Scene Complexity on Colour Constancy with Real Three-Dimensional Scenes and Objects

The effect of scene complexity on colour constancy was tested with a novel technique in which a virtual image of a real 3-D test object was projected into a real 3-D scene. Observers made discriminations between illuminant and material changes in simple and complex scenes. The extent of colour constancy achieved varied little with either scene structure or test-object colour, suggesting a dominant role of local cues in determining surface-colour judgments.

Lightness constancy and its link with cone contrast

The link between chromatic constancy (compensation for hue and saturation shifts) and lightness constancy (compensation for a change in surface reflectance) was tested theoretically by computing cone contrasts and by asymmetric matching experiments. The effect of a thin achromatic line (a frame) around the test sample was tested empirically. When the samples were outlined by the frame, lightness constancy was increased and chromatic constancy reduced (p<0.001). Changes in luminance are more likely to be compensated when the luminance contrast edge around the test stimulus is disturbed as with the addition of an achromatic frame.

Effects of chromatic image statistics on illumination induced color differences

We measure the color fidelity of visual scenes that are rendered under different (simulated) illuminants and shown on a calibrated LCD display. Observers make triad illuminant comparisons involving the renderings from two chromatic test illuminants and one achromatic reference illuminant shown simultaneously. Four chromatic test illuminants are used: two along the daylight locus (yellow and blue), and two perpendicular to it (red and green). The observers select the rendering having the best color fidelity, thereby indirectly judging which of the two test illuminants induces the smallest color differences compared to the reference. Both multicolor test scenes and natural scenes are studied. The multicolor scenes are synthesized and represent ellipsoidal distributions in CIELAB chromaticity space having the same mean chromaticity but different chromatic orientations. We show that, for those distributions, color fidelity is best when the vector of the illuminant change (pointing from neutral to chromatic) is parallel to the major axis of the scene's chromatic distribution. For our selection of natural scenes, which generally have much broader chromatic distributions, we measure a higher color fidelity for the yellow and blue illuminants than for red and green. Scrambled versions of the natural images are also studied to exclude possible semantic effects. We quantitatively predict the average observer response (i.e., the illuminant probability) with four types of models, differing in the extent to which they incorporate information processing by the visual system. Results show different levels of performance for the models, and different levels for the multicolor scenes and the natural scenes. Overall, models based on the scene averaged color difference have the best performance. We discuss how color constancy algorithms may be improved by exploiting knowledge of the chromatic distribution of the visual scene.

Color constancy

A quarter of a century ago, the first systematic behavioral experiments were performed to clarify the nature of color constancy-the effect whereby the perceived color of a surface remains constant despite changes in the spectrum of the illumination. At about the same time, new models of color constancy appeared, along with physiological data on cortical mechanisms and photographic colorimetric measurements of natural scenes. Since then, as this review shows, there have been many advances. The theoretical requirements for constancy have been better delineated and the range of experimental techniques has been greatly expanded; novel invariant properties of images and a variety of neural mechanisms have been identified; and increasing recognition has been given to the relevance of natural surfaces and scenes as laboratory stimuli. Even so, there remain many theoretical and experimental challenges, not least to develop an account of color constancy that goes beyond deterministic and relatively simple laboratory stimuli and instead deals with the intrinsically variable nature of surfaces and illuminations present in the natural world.

Auditory color constancy: calibration to reliable spectral properties across nonspeech context and targets

Brief experience with reliable spectral characteristics of a listening context can markedly alter perception of subsequent speech sounds, and parallels have been drawn between auditory compensation for listening context and visual color constancy. In order to better evaluate such an analogy, the generality of acoustic context effects for sounds with spectral-temporal compositions distinct from speech was investigated. Listeners identified nonspeech sounds-extensively edited samples produced by a French horn and a tenor saxophone-following either resynthesized speech or a short passage of music. Preceding contexts were "colored" by spectral envelope difference filters, which were created to emphasize differences between French horn and saxophone spectra. Listeners were more likely to report hearing a saxophone when the stimulus followed a context filtered to emphasize spectral characteristics of the French horn, and vice versa. Despite clear changes in apparent acoustic source, the auditory system calibrated to relatively predictable spectral characteristics of filtered context, differentially affecting perception of subsequent target nonspeech sounds. This calibration to listening context and relative indifference to acoustic sources operates much like visual color constancy, for which reliable properties of the spectrum of illumination are factored out of perception of color.

Effect of Scene Dimensionality on Colour Constancy with Real Three-Dimensional Scenes and Objects

The effect of scene dimensionality on colour constancy was tested with real scenes and objects. Observers viewed a three-dimensional (3-D) scene, or its two-dimensional (2-D) planar projection, through a large beam-splitter that projected the virtual image of a real test object (a cube or its 2-D projection) so that it appeared part of the scene. Test object and scene could be illuminated independently with high chromatic precision. In each trial, the illuminance of the scene changed abruptly from 25 000 K to 6700 K and the illuminant of the test object changed either consistently or inconsistently with it by a variable quantifiable amount. Observers had to decide whether the test object underwent a change in its materials. The extent of constancy obtained in the experiment was not influenced by scene dimensionality and varied significantly with the colour of the test object. These results suggest that color constancy in the conditions tested here may be determined by local spectral quantities.

Perception of Illuminant Colour Changes across Real Scenes

In a complex natural scene the colour and intensity of the illumination may vary considerably across the scene. Changes in intensity can easily be detected but the same does not seem to be true of colour changes. We investigated the extent to which chromatic changes of the illuminant are detected and the relation of detection performance with colour constancy and scene interpretation. The stimuli were complex real 3-D scenes rendered with spatial colour gradients of which the extremes had correlated colour temperatures within the range 25 000 K–3300 K. Observers' sensitivity to these spatial changes of the illuminant was found to be low and critically dependent on scene composition. Also, even in extreme conditions where colour constancy is known to fail, changes in the color of the illuminant across the scenes could not be perceived. These results suggest that insensitivity to spatial changes of the colour of the illuminant is a strong phenomenon and that it holds regardless of colour constancy.

Color constancy: phenomenal or projective?

Naive observers viewed a sequence of colored Mondrian patterns, simulated on a color monitor. Each pattern was presented twice in succession, first under one daylight illuminant with a correlated color temperature of either 16,000 or 4000 K and then under the other, to test for color constancy. The observers compared the central square of the pattern across illuminants, either rating it for sameness of material appearance or sameness of hue and saturation or judging an objective property-that is, whether its change of color originated from a change in material or only from a change in illumination. Average color constancy indices were high for material appearance ratings and binary judgments of origin and low for hue-saturation ratings. Individuals' performance varied, but judgments of material and of hue and saturation remained demarcated. Observers seem able to separate phenomenal percepts from their ontological projections of mental appearance onto physical phenomena; thus, even when a chromatic change alters perceived hue and saturation, observers can reliably infer the cause, the constancy of the underlying surface spectral reflectance.

Color constancy in natural scenes explained by global image statistics

To what extent do observers' judgments of surface color with natural scenes depend on global image statistics? To address this question, a psychophysical experiment was performed in which images of natural scenes under two successive daylights were presented on a computer-controlled high-resolution color monitor. Observers reported whether there was a change in reflectance of a test surface in the scene. The scenes were obtained with a hyperspectral imaging system and included variously trees, shrubs, grasses, ferns, flowers, rocks, and buildings. Discrimination performance, quantified on a scale of 0 to 1 with a color-constancy index, varied from 0.69 to 0.97 over 21 scenes and two illuminant changes, from a correlated color temperature of 25,000 K to 6700 K and from 4000 K to 6700 K. The best account of these effects was provided by receptor-based rather than colorimetric properties of the images. Thus, in a linear regression, 43% of the variance in constancy index was explained by the log of the mean relative deviation in spatial cone-excitation ratios evaluated globally across the two images of a scene. A further 20% was explained by including the mean chroma of the first image and its difference from that of the second image and a further 7% by the mean difference in hue. Together, all four global color properties accounted for 70% of the variance and provided a good fit to the effects of scene and of illuminant change on color constancy, and, additionally, of changing test-surface position. By contrast, a spatial-frequency analysis of the images showed that the gradient of the luminance amplitude spectrum accounted for only 5% of the variance.

Cone contrasts do not predict color constancy

A successive, asymmetric color-matching paradigm was used to investigate the link between cone contrast and the stability of perceived colors. We measured the perceived color shifts of 10 Munsell samples, induced by test illuminant A, simulated in u'v' color space. The capacity of the visual system to resist these shifts, otherwise known as color constancy, is measured in terms of the Brunswik ratio, BR. Cone contrasts are calculated with respect to either the physical or perceived background. Subjective cone contrasts show a better fit to the von Kries law than those based on the physical background. Complete cone adaptation occurs when color constancy is high. However we show conditions where cone adaptation seems complete but color constancy is poor.

Color constancy in natural scenes with and without an explicit illuminant cue

Observers can generally make reliable judgments of surface color in natural scenes despite changes in an illuminant that is out of view. This ability has sometimes been attributed to observers' estimating the spectral properties of the illuminant in order to compensate for its effects. To test this hypothesis, two surface-color-matching experiments were performed with images of natural scenes obtained from high-resolution hyperspectral images. In the first experiment, the sky illuminating the scene was directly visible to the observer, and its color was manipulated. In the second experiment, a large gray sphere was introduced into the scene so that its illumination by the sun and sky was also directly visible to the observer, and the color of that illumination was manipulated. Although the degree of color constancy varied across this and other variations of the images, there was no reliable effect of illuminant color. Even when the sky was eliminated from view, color constancy did not worsen. Judging surface color in natural scenes seems to be independent of an explicit illuminant cue.

Minimalist surface-colour matching

Some theories of surface-colour perception assume that observers estimate the illuminant on a scene so that its effects can be discounted. A critical test of this interpretation of colour constancy is whether surface-colour matching is worse when the number of surfaces in a scene is so small that any illuminant estimate is unreliable. In the experiment reported here, observers made asymmetric colour matches between pairs of simultaneously presented Mondrian-like patterns under different daylights. The patterns had either 49 surfaces or a minimal 2 surfaces. No significant effect of number was found, suggesting that illuminant estimates are unnecessary for surface-colour matching.

Does colour constancy exist?

For a stable visual world, the colours of objects should appear the same under different lights. This property of colour constancy has been assumed to be fundamental to vision, and many experimental attempts have been made to quantify it. I contend here, however, that the usual methods of measurement are either too coarse or concentrate not on colour constancy itself, but on other, complementary aspects of scene perception. Whether colour constancy exists other than in nominal terms remains unclear.

Minimum-variance cone-excitation ratios and the limits of relational color constancy

Relational color constancy refers to the constancy of the perceived relations between the colors of surfaces of a scene under changes in the spectral composition of the illuminant. Spatial ratios of cone excitations provide a natural physical basis for this constancy, as, on average, they are almost invariant under illuminant changes for large collections of natural surfaces and illuminants. The aim of the present work was to determine, computationally, for specific surfaces and illuminants, the constancy limits obtained by the application of a minimum-variance principle to cone-excitation ratios and to investigate its validity in predicting observers' surface-color judgments. Cone excitations and their changes due to variations in the color of the illuminant were estimated for colored surfaces in simulated two-dimensional scenes of colored papers and real three-dimensional scenes of solid colored objects. For various test surfaces, scenes, and illuminants, the estimated levels of relational color constancy mediated by cone-excitation ratios varied significantly with the test surface and only with certain desaturated surfaces corresponded to ideal matches. Observers' experimental matches were compared with predictions expressed in CIE 1976 (u′,v′) space and were found to be generally consistent with minimum-variance predictions.

Color constancy by asymmetric color matching with real objects in three-dimensional scenes

Color matching experiments use, in general, stimuli that are poor representations of the natural world. The aim of this work was to compare the degree of color constancy for a range of illuminant pairs using a new matching technique that uses both real objects and three-dimensional (3-D) real scenes. In the experiment, observers viewed a 3-D real scene through a large beamsplitter that projects on the right-hand side of the scene (match scene), the virtual image of a 3-D object (match object) such it appeared part of the scene. On the left-hand side of the scene (test scene), observers viewed a symmetrical scene containing a test object identical to the match object. Test and match objects were both surrounded by the same reflectances with identical spatial arrangement. The illuminant on the test scene had always a correlated color temperature of 25,000 K. The illuminant on the match scene could be any of seven different illuminants with correlated color temperatures in the range 25,000 K–4000 K. In each trial, the observers, who were instructed to perform surface color matches, adjusted the illuminant on the match object. Constancy indices were very high (0.81–0.93), varied with the color of the match object, and increased with the extent of the illuminant change. Observer's mismatches, however, were independent of the extent of the illuminant change.

Colour constancy under simultaneous changes in surface position and illuminant

Two kinds of constancy underlie the everyday perception of surface colour: constancy under changes in illuminant and constancy under changes in surface position. Classically, these two constancies seem to place conflicting demands on the visual system: to both take into account the region surrounding a surface and also discount it. It is shown here, however, that the ability of observers to make surface-colour matches across simultaneous changes in test-surface position and illuminant in computer-generated 'Mondrian' patterns is almost as good as across changes in illuminant alone. Performance was no poorer when the surfaces surrounding the test surface were permuted, or when information from a potential comparison surface, the one with the highest luminance, was suppressed. Computer simulations of cone-photoreceptor activity showed that a reliable cue for making surface-colour matches in all experimental conditions was provided by the ratios of cone excitations between the test surfaces and a spatial average over the whole pattern.

Statistics of spatial cone-excitation ratios in natural scenes

For some sets of surfaces, the spatial ratios of cone-photoreceptor excitations produced by light reflected from pairs of surfaces are almost invariant under illuminant changes. These sets include large populations of spectral reflectances, some of which represent individual natural surfaces but not their relative abundances in nature. The aim of this study was to determine whether spatial cone-excitation ratios are preserved under illuminant changes within the natural visual environment. A fast hyperspectral imaging system was used to obtain populations of 640,000 reflectance spectra from each of 30 natural scenes. The statistics of spatial cone-excitation ratios for randomly selected pairs of points in these scenes were determined for two extreme daylights. Almost-invariant ratios were common, suggesting that they represent a reliable property of the natural visual environment and a suitable foundation for visual color constancy.

Detecting changes of spatial cone-excitation ratios in dichoptic viewing

Spatial ratios of cone excitations produced by light reflected by different surfaces in a scene may provide the cue for discriminating changes in illuminant from changes in surface reflectances. To test whether these ratios can be computed across the two eyes, observers were presented with simulations on a computer-controlled monitor of pairs of juxtaposed or separated Munsell surfaces undergoing an illuminant change with a small change in cone-excitation ratios or a change with constant cone-excitation ratios. Surfaces were viewed either binocularly or dichoptically. Observers reliably discriminated the two changes in both viewing conditions, although less well dichoptically. Cone-excitation ratios, which may in principle be computed retinally, may also be computed cortically.

Parallel detection of violations of color constancy

The perceived colors of reflecting surfaces generally remain stable despite changes in the spectrum of the illuminating light. This color constancy can be measured operationally by asking observers to distinguish illuminant changes on a scene from changes in the reflecting properties of the surfaces comprising it. It is shown here that during fast illuminant changes, simultaneous changes in spectral reflectance of one or more surfaces in an array of other surfaces can be readily detected almost independent of the numbers of surfaces, suggesting a preattentive, spatially parallel process. This process, which is perfect over a spatial window delimited by the anatomical fovea, may form an early input to a multistage analysis of surface color, providing the visual system with information about a rapidly changing world in advance of the generation of a more elaborate and stable perceptual representation.