Color constancy in a naturalistic, goal-directed task

Color constancy mechanisms in virtual reality environments

Prior research has demonstrated high levels of color constancy in real-world scenarios featuring single light sources, extensive fields of view, and prolonged adaptation periods. However, exploring the specific cues humans rely on becomes challenging, if not unfeasible, with actual objects and lighting conditions. To circumvent these obstacles, we employed virtual reality technology to craft immersive, realistic settings that can be manipulated in real time. We designed forest and office scenes illuminated by five colors. Participants selected a test object most resembling a previously shown achromatic reference. To study color constancy mechanisms, we modified scenes to neutralize three contributors: local surround (placing a uniform-colored leaf under test objects), maximum flux (keeping the brightest object constant), and spatial mean (maintaining a neutral average light reflectance), employing two methods for the latter: changing object reflectances or introducing new elements. We found that color constancy was high in conditions with all cues present, aligning with past research. However, removing individual cues led to varied impacts on constancy. Local surrounds significantly reduced performance, especially under green illumination, showing strong interaction between greenish light and rose-colored contexts. In contrast, the maximum flux mechanism barely affected performance, challenging assumptions used in white balancing algorithms. The spatial mean experiment showed disparate effects: Adding objects slightly impacted performance, while changing reflectances nearly eliminated constancy, suggesting human color constancy relies more on scene interpretation than pixel-based calculations.

Color constancy in real-world settings

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

The Perceptual Science of Augmented Reality

Augmented reality (AR) systems aim to alter our view of the world and enable us to see things that are not actually there. The resulting discrepancy between perception and reality can create compelling entertainment and can support innovative approaches to education, guidance, and assistive tools. However, building an AR system that effectively integrates with our natural visual experience is hard. AR systems often suffer from visual limitations and artifacts, and addressing these flaws requires basic knowledge of perception. At the same time, AR system development can serve as a catalyst that drives innovative new research in perceptual science. This review describes recent perceptual research pertinent to and driven by modern AR systems, with the goal of highlighting thought-provoking areas of inquiry and open questions.

Colour Calibration of a Head Mounted Display for Colour Vision Research Using Virtual Reality

Virtual reality (VR) technology offers vision researchers the opportunity to conduct immersive studies in simulated real-world scenes. However, an accurate colour calibration of the VR head mounted display (HMD), both in terms of luminance and chromaticity, is required to precisely control the presented stimuli. Such a calibration presents significant new challenges, for example, due to the large field of view of the HMD, or the software implementation used for scene rendering, which might alter the colour appearance of objects. Here, we propose a framework for calibrating an HMD using an imaging colorimeter, the I29 (Radiant Vision Systems, Redmond, WA, USA). We examine two scenarios, both with and without using a rendering software for visualisation. In addition, we present a colour constancy experiment design for VR through a gaming engine software, Unreal Engine 4. The colours of the objects of study are chosen according to the previously defined calibration. Results show a high-colour constancy performance among participants, in agreement with recent studies performed on real-world scenarios. Our studies show that our methodology allows us to control and measure the colours presented in the HMD, effectively enabling the use of VR technology for colour vision research.

The effect of scene articulation on transparent layer constancy

In this article, we examine the influence of scene articulation on transparent layer constancy. We argue that the term articulation may be understood as an aspect of the more general concept naturalness of a stimulus that relates to the degree of enrichment compared with a minimal stimulus and to the extent to which a stimulus contains regularities that are typically found in natural scenes. We conducted two matching experiments, in which we used strongly reduced scenes and operationalized articulation by the number of background reflectances (numerosity). The results of the first experiment show that higher numerosity actually leads to an increase in transparent layer constancy when reflectances are randomly drawn from a fixed population. However, this advantage disappears if the spatial mean and the variation of the subset colors are controlled as in our second experiment. Furthermore, our results suggest that the mechanism underlying transparent layer constancy leads to a rather stable compromise between two matching criteria, namely, proximal identity and constant filter properties according to our perceptual model. For filters with an additive component, which appear more or less hazy, we observed improved recovered filter properties and correspondingly higher degrees of transparent layer constancy, suggesting an additional mechanism in this type of filter.

Temporal dynamics of daylight perception: Detection thresholds

Temporal changes in illumination are ubiquitous; natural light, for example, varies in color temperature and irradiance throughout the day. Yet little is known about human sensitivity to temporal changes in illumination spectra. Here, we aimed to determine the minimum detectable velocity of chromaticity change of daylight metamers in an immersive environment. The main stimulus was a continuous, monotonic change in global illumination chromaticity along the daylight locus in warmer (toward lower correlated color temperatures [CCTs]) or cooler directions, away from an adapting base light (CCT: 13,000 K, 6500 K, 4160 K, or 2000 K). All lights were generated by spectrally tunable overhead lamps as smoothest-possible metamers of the desired chromaticities. Mean detection thresholds (for 22 participants) for a fixed duration of 10 seconds ranged from 15 to 2 CIELUV ΔE units, depending significantly on base light CCT and with a significant interaction between CCT and direction of change. Cool changes become less noticeable for progressively warmer base lights and vice versa. For the two extreme base lights, sensitivity to changes toward neutral is significantly lower than for the opposite direction. The results suggest a “neutral bias” in illumination change discriminability, and that typical temporal changes in daylight chromaticity are likely to be below threshold detectability, at least where there are no concomitant overall illuminance changes. These factors may contribute to perceptual stability of natural scenes and color constancy.

Measuring and Modeling Visual Appearance

In studying visual perception, we seek to develop models of processing that accurately predict perceptual judgments. Much of this work is focused on judgments of discrimination, and there is a large literature concerning models of visual discrimination. There are, however, non-threshold visual judgments, such as judgments of the magnitude of differences between visual stimuli, that provide a means to bridge the gap between threshold and appearance. We describe two such models of suprathreshold judgments, maximum likelihood difference scaling and maximum likelihood conjoint measurement, and review recent literature that has exploited them.

Transparent layer constancy is improved by motion, stereo disparity, highly regular background pattern, and successive presentation

The visual system uses figural and colorimetric regularities in the retinal image to recognize optical filters and to discern the properties of the transparent overlay from properties of the background. Previous work suggests that the perceived color and transmittance of the transparent layer vary less under illumination changes than it would be expected from corresponding changes in the input. Here, we tested how the degree of this approximate transparent layer constancy (TLC) depends on factors that presumably facilitate the decomposition into a filter and a background layer. Using an asymmetric filter matching task, we found that motion, stereo disparity, and a highly regular background pattern each contribute to the vividness of the transparency impression and the degree of TLC. Combining these cues led to a cumulative increase in TLC, suggesting a "strong fusion" cue integration process. We also tested objects with invalid figural conditions for transparency (T-junctions). The tendency to perceive these objects as opaque and to establish a proximal match increased the more conspicuous the violation of this figural condition was. Furthermore, we investigated the gain in TLC due to alternating presentation. Alternating presentation enhanced TLC and color constancy to a comparable degree, and our results suggest that adaptation contributes to this effect.

The relative contribution of color and material in object selection

Object perception is inherently multidimensional: information about color, material, texture and shape all guide how we interact with objects. We developed a paradigm that quantifies how two object properties (color and material) combine in object selection. On each experimental trial, observers viewed three blob-shaped objects—the target and two tests—and selected the test that was more similar to the target. Across trials, the target object was fixed, while the tests varied in color (across 7 levels) and material (also 7 levels, yielding 49 possible stimuli). We used an adaptive trial selection procedure (Quest+) to present, on each trial, the stimulus test pair that is most informative of underlying processes that drive selection. We present a novel computational model that allows us to describe observers’ selection data in terms of (1) the underlying perceptual stimulus representation and (2) a color-material weight, which quantifies the relative importance of color vs. material in selection. We document large individual differences in the color-material weight across the 12 observers we tested. Furthermore, our analyses reveal limits on how precisely selection data simultaneously constrain perceptual representations and the color-material weight. These limits should guide future efforts towards understanding the multidimensional nature of object perception.

Much is known about how the visual system extracts information about individual object properties, such as color or material. Considerably less is known about how percepts of these properties interact to form a multidimensional object representation. We report the first quantitative analysis of how perceived color and material combine in object selection, using a task designed to reflect key aspects of how we use vision in real life. We introduce a computational model that describes observers’ selection behavior in terms of (1) how objects are represented in an underlying subjective perceptual color-material space and (2) how differences in perceived object color and material combine to guide selection. We find large individual differences in the degree to which observers select objects based on color relative to material: some base their selections almost entirely on color, some weight color and material nearly equally, and others rely almost entirely on material. A fine-grained analysis clarifies the limits on how precisely selection data may be leveraged to simultaneously understand the underlying perceptual representations on one hand and how the information about perceived color and material combine on the other. Our work provides a foundation for improving our understanding of visual computations in natural viewing.

Illumination discrimination for chromatically biased illuminations: Implications for color constancy

We measured discrimination thresholds for illumination changes along different chromatic directions starting from chromatically biased reference illuminations. Participants viewed a Mondrian-papered scene illuminated by LED lamps. The scene was first illuminated by a reference illumination, followed by two comparisons. One comparison matched the reference (the target); the other (the test) varied from the reference, nominally either bluer, yellower, redder, or greener. The participant's task was to correctly select the target. A staircase procedure found thresholds for discrimination of an illumination change along each axis of chromatic change. Nine participants completed the task for five different reference illumination conditions (neutral, blue, yellow, red, and green). We find that relative discrimination thresholds for different chromatic directions of illumination change vary with the reference illumination. For the neutral reference, there is a trend for thresholds to be highest in the bluer illumination-change direction, replicating our previous reports of a "blue bias" for neutral reference illuminations. For the four chromatic references (blue, yellow, red, and green), the change in illumination toward the neutral reference is less well discriminated than changes in the other directions: a "neutral bias." The results have implications for color constancy: In considering the stability of surface appearance under changes in illumination, both the starting chromaticity of the illumination and direction of change must be considered, as well as the chromatic characteristics of the surface reflectance ensemble. They also suggest it will be worthwhile to explore whether and how the human visual system has internalized the statistics of natural illumination changes.

Color Perception: Objects, Constancy, and Categories

Color has been scientifically investigated by linking color appearance to colorimetric measurements of the light that enters the eye. However, the main purpose of color perception is not to determine the properties of incident light, but to aid the visual perception of objects and materials in our environment. We review the state of the art on object colors, color constancy, and color categories to gain insight into the functional aspects of color perception. The common ground between these areas of research is that color appearance is tightly linked to the identification of objects and materials and the communication across observers. In conclusion, we argue that research should focus on how color processing is adapted to the surface properties of objects in the natural environment in order to bridge the gap between the known early stages of color perception and the subjective appearance of color.

The perception of colour and material in naturalistic tasks

Perceived object colour and material help us to select and interact with objects. Because there is no simple mapping between the pattern of an object's image on the retina and its physical reflectance, our perceptions of colour and material are the result of sophisticated visual computations. A long-standing goal in vision science is to describe how these computations work, particularly as they act to stabilize perceived colour and material against variation in scene factors extrinsic to object surface properties, such as the illumination. If we take seriously the notion that perceived colour and material are useful because they help guide behaviour in natural tasks, then we need experiments that measure and models that describe how they are used in such tasks. To this end, we have developed selection-based methods and accompanying perceptual models for studying perceived object colour and material. This focused review highlights key aspects of our work. It includes a discussion of future directions and challenges, as well as an outline of a computational observer model that incorporates early, known, stages of visual processing and that clarifies how early vision shapes selection performance.

Discrimination of spectral reflectance under environmental illumination

Color constancy is the ability to recover a stable perceptual estimate of surface reflectance, regardless of the lighting environment. However, we know little about how observers make judgments of the surface color of glossy objects, particularly in complex lighting environments that introduce complex spatial patterns of chromatic variation across an object's surface. To address this question, we measured thresholds for reflectance discrimination using computer-rendered stimuli under environmental illumination. In Experiment 1, we found that glossiness and shape had small effects on discrimination thresholds. Importantly, discrimination ellipses extended along the direction in which the chromaticities in the environmental illumination spread. In Experiment 2, we also found that the observers' abilities to judge surface colors were worse in lighting environments with an atypical chromatic distribution.

Chips in the sunshine: color constancy with real versus simulated Munsell chips under illuminants adjacent to the daylight locus

Accurate color judgments rely on a powerful cognitive component. Here we compare the performance of color constancy under real and simulated conditions. Shifts in the u^'v^' color plane induced by illuminant A (2750 K) and illuminant S (>20,000  K) were measured using asymmetric color matching. A general linear model was used to predict performance from the following dependent variables: chroma ("4" and "6"), illuminant ("A" and "S"), presentation mode ("Real" and "Monitor"), and hue zone ("blue," "green," "yellow," "red," and "purple"). There was a strong overall effect [F(7,264)=78.65, p<0.001]. Post hoc analysis showed that performance was substantially superior under real [chromatic constancy index (cCI)=0.76] compared with simulated cCI=0.55) conditions.

Kitaoka's Tomato: Two Simple Explanations Based on Information in the Stimulus

Kitaoka’s Tomato is a color illusion in which a semitransparent blue-green field is placed on top of a red object (a tomato). The tomato appears red even though the pixels would appear green if viewed in isolation. We show that this phenomenon can be explained by a high-pass filter and by histogram equalization. The results suggest that this illusion does not require complex inferences about color constancy; rather, the tomato’s red is available in the physical stimulus at the appropriate spatial scale and dynamic range.

What #theDress reveals about the role of illumination priors in color perception and color constancy

The disagreement between people who named #theDress (the Internet phenomenon of 2015) "blue and black" versus "white and gold" is thought to be caused by individual differences in color constancy. It is hypothesized that observers infer different incident illuminations, relying on illumination "priors" to overcome the ambiguity of the image. Different experiences may drive the formation of different illumination priors, and these may be indicated by differences in chronotype. We assess this hypothesis, asking whether matches to perceived illumination in the image and/or perceived dress colors relate to scores on the morningness-eveningness questionnaire (a measure of chronotype). We find moderate correlations between chronotype and illumination matches (morning types giving bluer illumination matches than evening types) and chronotype and dress body matches, but these are significant only at the 10% level. Further, although inferred illumination chromaticity in the image explains variation in the color matches to the dress (confirming the color constancy hypothesis), color constancy thresholds obtained using an established illumination discrimination task are not related to dress color perception. We also find achromatic settings depend on luminance, suggesting that subjective white point differences may explain the variation in dress color perception only if settings are made at individually tailored luminance levels. The results of such achromatic settings are inconsistent with their assumed correspondence to perceived illumination. Finally, our results suggest that perception and naming are disconnected, with observers reporting different color names for the dress photograph and their isolated color matches, the latter best capturing the variation in the matches.

The nature of instructional effects in color constancy

The instructions subjects receive can have a large effect on experimentally measured color constancy, but the nature of these effects and how their existence should inform our understanding of color perception remains unclear. We used a factorial design to measure how instructional effects on constancy vary with experimental task and stimulus set. In each of 2 experiments, we employed both a classic adjustment-based asymmetric matching task and a novel color selection task. Four groups of naive subjects were instructed to make adjustments/selections based on (a) color (neutral instructions); (b) the light reaching the eye (physical spectrum instructions); (c) the actual surface reflectance of an object (objective reflectance instructions); or (d) the apparent surface reflectance of an object (apparent reflectance instructions). Across the 2 experiments we varied the naturalness of the stimuli. We find clear interactions between instructions, task, and stimuli. With simplified stimuli (Experiment 1), instructional effects were large and the data revealed 2 instruction-dependent patterns. In 1 (neutral and physical spectrum instructions) constancy was low, intersubject variability was also low, and adjustment-based and selection-based constancy were in agreement. In the other (reflectance instructions) constancy was high, intersubject variability was large, adjustment-based constancy deviated from selection-based constancy and for some subjects selection-based constancy increased across sessions. Similar patterns held for naturalistic stimuli (Experiment 2), although instructional effects were smaller. We interpret these 2 patterns as signatures of distinct task strategies-1 is perceptual, with judgments based primarily on the perceptual representation of color; the other involves explicit instruction-driven reasoning. (PsycINFO Database Record

Color constancy supports cross-illumination color selection

We rely on color to select objects as the targets of our actions (e.g., the freshest fish, the ripest fruit). To be useful for selection, color must provide accurate guidance about object identity across changes in illumination. Although the visual system partially stabilizes object color appearance across illumination changes, how such color constancy supports object selection is not understood. To study how constancy operates in real-life tasks, we developed a novel paradigm in which subjects selected which of two test objects presented under a test illumination appeared closer in color to a target object presented under a standard illumination. From subjects' choices, we inferred a selection-based match for the target via a variant of maximum likelihood difference scaling, and used it to quantify constancy. Selection-based constancy was good when measured using naturalistic stimuli, but was dramatically reduced when the stimuli were simplified, indicating that a naturalistic stimulus context is critical for good constancy. Overall, our results suggest that color supports accurate object selection across illumination changes when both stimuli and task match how color is used in real life. We compared our selection-based constancy results with data obtained using a classic asymmetric matching task and found that the adjustment-based matches predicted selection well for our stimuli and instructions, indicating that the appearance literature provides useful guidance for the emerging study of constancy in natural tasks.