The color constancy of three-dimensional objects

Color and gloss constancy under diverse lighting environments

When we look at an object, we simultaneously see how glossy or matte it is, how light or dark, and what color. Yet, at each point on the object's surface, both diffuse and specular reflections are mixed in different proportions, resulting in substantial spatial chromatic and luminance variations. To further complicate matters, this pattern changes radically when the object is viewed under different lighting conditions. The purpose of this study was to simultaneously measure our ability to judge color and gloss using an image set capturing diverse object and illuminant properties. Participants adjusted the hue, lightness, chroma, and specular reflectance of a reference object so that it appeared to be made of the same material as a test object. Critically, the two objects were presented under different lighting environments. We found that hue matches were highly accurate, except for under a chromatically atypical illuminant. Chroma and lightness constancy were generally poor, but these failures correlated well with simple image statistics. Gloss constancy was particularly poor, and these failures were only partially explained by reflection contrast. Importantly, across all measures, participants were highly consistent with one another in their deviations from constancy. Although color and gloss constancy hold well in simple conditions, the variety of lighting and shape in the real world presents significant challenges to our visual system's ability to judge intrinsic material properties.

Perceiving the representative surface color of real-world materials

Natural surfaces such as soil, grass, and skin usually involve far more complex and heterogenous structures than the perfectly uniform surfaces assumed in studies on color and material perception. Despite this, we can easily perceive the representative color of these surfaces. Here, we investigated the visual mechanisms underlying the perception of representative surface color using 120 natural images of diverse materials and their statistically synthesized images. Our matching experiments indicated that the perceived representative color revealed was not significantly different from the Portilla-Simoncelli-synthesized images or phase-randomized images except for one sample, even though the perceived shape and material properties were greatly impaired in the synthetic stimuli. The results also showed that the matched representative colors were predictable from the saturation-enhanced color of the brightest point in the image, excluding the high-intensity outliers. The results support the notion that humans judge the representative color and lightness of real-world surfaces depending on simple image measurements.

The effect of surface gloss on categorical color constancy in real scenes

Categorical color constancy has been widely investigated and found to be very robust. As one of object material properties, the surface gloss was found to barely contribute to color constancy in a natural viewing condition. In this study, the effect of surface gloss on categorical color constancy was investigated by asking eight observers to categorize 208 Munsell matte surfaces and 260 Munsell glossy surfaces under D65, F, and TL84 illuminants in a viewing chamber with a uniform gray background. A color constancy index based on the centroid shift of the color category was used to evaluate color constancy degree of each color category across illumination changes from D65 to F or TL84 illuminant. The result showed that both matte and glossy surfaces showed almost perfect color constancy on all color categories under F and TL84 illuminants, and there was no significant difference between them. This result suggests that surface gloss has little effect on categorical color constancy in a uniform gray background where the local surround cue was present, which is consistent with the previous findings.

The dominating impacts of Helmholtz-Kohlrausch effect on color-induced glossiness enhancement

Glossiness can be increased by adding chromatic information to the object images. However, the mechanisms that create color-induced glossiness enhancement are unclear. In this study, we psychophysically measured the glossiness of object images to which various hue chromaticities were added to elucidate the perceptual and image factors that explain the color-induced glossiness enhancement effect. Two types of coloring conditions were tested: the both-colored (BC) condition, in which both specular and diffuse components were colored with the same chromaticity, and the diffuse-colored (DC) condition, in which only diffuse components were colored while specular components remained achromatic. The results showed that glossiness enhancement was more prominent in the BC than in the DC condition, and the dependency of glossiness enhancement on the stimulus color direction was similar to that of the Helmholtz-Kohlrausch (H-K) effect. Furthermore, we performed a regression analysis with a linear mixed model based on image features and an additional experiment in which an H-K effect-based increase in perceived brightness was imitated on achromatic stimuli by manipulating luminance. The results demonstrated that the H-K effect-based brightness enhancement in the highlight regions explains the glossiness enhancement effect well. These results suggest that the H-K effect, especially around the highlight region, is a dominant factor that creates the color-induced glossiness enhancement, although other color-related factors could also be partly involved.

The effect of scene articulation on transparent layer constancy

In this article, we examine the influence of scene articulation on transparent layer constancy. We argue that the term articulation may be understood as an aspect of the more general concept naturalness of a stimulus that relates to the degree of enrichment compared with a minimal stimulus and to the extent to which a stimulus contains regularities that are typically found in natural scenes. We conducted two matching experiments, in which we used strongly reduced scenes and operationalized articulation by the number of background reflectances (numerosity). The results of the first experiment show that higher numerosity actually leads to an increase in transparent layer constancy when reflectances are randomly drawn from a fixed population. However, this advantage disappears if the spatial mean and the variation of the subset colors are controlled as in our second experiment. Furthermore, our results suggest that the mechanism underlying transparent layer constancy leads to a rather stable compromise between two matching criteria, namely, proximal identity and constant filter properties according to our perceptual model. For filters with an additive component, which appear more or less hazy, we observed improved recovered filter properties and correspondingly higher degrees of transparent layer constancy, suggesting an additional mechanism in this type of filter.

Surface properties and the perception of color

We examined whether perception of color saturation and lightness depends on the three-dimensional (3D) shape and surface gloss of surfaces rendered to have different hues. In Experiment 1, we parametrically varied specular roughness of predominantly planar surfaces with different mesoscopic relief heights. The orientation of surfaces was varied relative to the light source and observer. Observers matched perceived lightness and chroma (effectively saturation) using spherical objects rendered using CIE LCH color space. We observed strong interactions between perceived saturation and lightness with changes in surface orientation and surface properties (specular roughness and 3D relief height). Declines in saturation and increases in lightness were observed with increasing specular roughness. Changes in relief height had greater effects on perceived saturation and lightness for blue hues compared with reddish and greenish hues. Experiment 2 found inverse correlations between perceived gloss and specular roughness across conditions. Experiment 3 estimated perceived specular coverage and found that a weighted combination of perceived gloss and specular coverage could account for perceived color saturation and lightness, with different coefficients accounting for the perceptual experience for each of the three hue conditions. These findings suggest that perceived color saturation and lightness depend on the separation of specular highlights from diffuse shading informative of chromatic surface reflectance.

Specular highlights improve color constancy when other cues are weakened

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination ("daylight priors"). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy. Observers made achromatic settings within three-dimensional rendered scenes containing matte or glossy shapes, illuminated by either daylight or nondaylight illuminations. We assessed both the variability of these settings and their accuracy, in terms of the standard color constancy index (CCI). When a spectrally uniform background was present, neither CCIs nor variability improved with specular highlights or daylight illuminants (Experiment 1). When a Mondrian background was introduced, CCIs decreased overall but were higher for scenes containing glossy, as opposed to matte, shapes (Experiments 2 and 3). There was no overall reduction in variability of settings and no benefit for scenes illuminated by daylights. Taken together, these results suggest that the human visual system indeed uses specular highlights to improve color constancy but only when other cues, such as from the local surround, are weakened.

Computational luminance constancy from naturalistic images

The human visual system supports stable percepts of object color even though the light that reflects from object surfaces varies significantly with the scene illumination. To understand the computations that support stable color perception, we study how estimating a target object's luminous reflectance factor (LRF; a measure of the light reflected from the object under a standard illuminant) depends on variation in key properties of naturalistic scenes. Specifically, we study how variation in target object reflectance, illumination spectra, and the reflectance of background objects in a scene impact estimation of a target object's LRF. To do this, we applied supervised statistical learning methods to the simulated excitations of human cone photoreceptors, obtained from labeled naturalistic images. The naturalistic images were rendered with computer graphics. The illumination spectra of the light sources and the reflectance spectra of the surfaces in the scene were generated using statistical models of natural spectral variation. Optimally decoding target object LRF from the responses of a small learned set of task-specific linear receptive fields that operate on a contrast representation of the cone excitations yields estimates that are within 13% of the correct LRF. Our work provides a framework for evaluating how different sources of scene variability limit performance on luminance constancy.

Color Perception: Objects, Constancy, and Categories

Color has been scientifically investigated by linking color appearance to colorimetric measurements of the light that enters the eye. However, the main purpose of color perception is not to determine the properties of incident light, but to aid the visual perception of objects and materials in our environment. We review the state of the art on object colors, color constancy, and color categories to gain insight into the functional aspects of color perception. The common ground between these areas of research is that color appearance is tightly linked to the identification of objects and materials and the communication across observers. In conclusion, we argue that research should focus on how color processing is adapted to the surface properties of objects in the natural environment in order to bridge the gap between the known early stages of color perception and the subjective appearance of color.

Discrimination of spectral reflectance under environmental illumination

Color constancy is the ability to recover a stable perceptual estimate of surface reflectance, regardless of the lighting environment. However, we know little about how observers make judgments of the surface color of glossy objects, particularly in complex lighting environments that introduce complex spatial patterns of chromatic variation across an object's surface. To address this question, we measured thresholds for reflectance discrimination using computer-rendered stimuli under environmental illumination. In Experiment 1, we found that glossiness and shape had small effects on discrimination thresholds. Importantly, discrimination ellipses extended along the direction in which the chromaticities in the environmental illumination spread. In Experiment 2, we also found that the observers' abilities to judge surface colors were worse in lighting environments with an atypical chromatic distribution.

Color Constancy in Two-Dimensional and Three-Dimensional Scenes: Effects of Viewing Methods and Surface Texture

There has been debate about how and why color constancy may be better in three-dimensional (3-D) scenes than in two-dimensional (2-D) scenes. Although some studies have shown better color constancy for 3-D conditions, the role of specific cues remains unclear. In this study, we compared color constancy for a 3-D miniature room (a real scene consisting of actual objects) and 2-D still images of that room presented on a monitor using three viewing methods: binocular viewing, monocular viewing, and head movement. We found that color constancy was better for the 3-D room; however, color constancy for the 2-D image improved when the viewing method caused the scene to be perceived more like a 3-D scene. Separate measurements of the perceptual 3-D effect of each viewing method also supported these results. An additional experiment comparing a miniature room and its image with and without texture suggested that surface texture of scene objects contributes to color constancy.

Motion of glossy objects does not promote separation of lighting and surface colour

The surface properties of an object, such as texture, glossiness or colour, provide important cues to its identity. However, the actual visual stimulus received by the eye is determined by both the properties of the object and the illumination. We tested whether operational colour constancy for glossy objects (the ability to distinguish changes in spectral reflectance of the object, from changes in the spectrum of the illumination) was affected by rotational motion of either the object or the light source. The different chromatic and geometric properties of the specular and diffuse reflections provide the basis for this discrimination, and we systematically varied specularity to control the available information. Observers viewed animations of isolated objects undergoing either lighting or surface-based spectral transformations accompanied by motion. By varying the axis of rotation, and surface patterning or geometry, we manipulated: (i) motion-related information about the scene, (ii) relative motion between the surface patterning and the specular reflection of the lighting, and (iii) image disruption caused by this motion. Despite large individual differences in performance with static stimuli, motion manipulations neither improved nor degraded performance. As motion significantly disrupts frame-by-frame low-level image statistics, we infer that operational constancy depends on a high-level scene interpretation, which is maintained in all conditions.

Facial Color Management for Mobile Health in the Wild

Nowadays, mobile technologies have changed the patient routine health care and management. With a large amount of mobile health applications developed, massive and valuable health data are possibly collected with a smart mobile phone in hand. Facial color images are recently proved to be available and effective for health condition diagnosis both in modern medicine and ancient medicine perspectives. One significant issue of facial color health condition diagnosis system is color management, in which its primary procedure is to obtain reliable and device-independent facial color images in the wild. The solution is known as utilizing color correction technology to recover the intrinsic color properties of facial skin. However, current color correction approaches are hard to meet the need of mobile health management in the wild, due to some limitations of precision-challenged algorithm, inconvenient color imaging device, strong scenario assumption and so forth. Therefore, in this paper, we consider several facial skin color characteristics and show that it is valuable to build facial color related correction model for facial color images in the wild. Then we propose two kinds of facial color correction strategies to realize the facial color management of mobile health in the wild. The first one is reference-based approach, and the other one is skin-based approach without requirement of colorchecker. Experimental results with qualitative and quantitative assessments on the indoor and outdoor scenarios demonstrate that the proposed reference-based approach is more outstanding than our previous method and other color constancy methods. In addition, given a facial color image only, the skin-based method can still achieve effective results compared with other color constancy methods.

Effects of surrounding stimulus properties on color constancy based on luminance balance

The visual system needs to discount the influence of an illuminant to achieve color constancy. Uchikawa et al. [J. Opt. Soc. Am. A29, A133 (2012) showed that the luminance-balance change of surfaces in a scene contributes to illuminant estimation; however, its effect was substantially less than the chromaticity change. We conduct three experiments to reinforce the previous findings and investigate possible factors that can influence the effect of luminance balance. Experimental results replicate the previous finding; i.e., luminance balance makes a small, but significant, contribution to illuminant estimation. We find that stimulus dimensionality affects neither the degree of color constancy nor the effect of luminance balance. Unlike chromaticity-based color constancy, chromatic variation does not influence the effect of luminance balance. It is shown that luminance-balance-based estimation of an illuminant performs better for scenes with reddish or bluish surfaces. This suggests that the visual system exploits the optimal color distribution for illuminant estimation [J. Opt. Soc. Am. A 29, A133(2012)].

Low levels of specularity support operational color constancy, particularly when surface and illumination geometry can be inferred

We tested whether surface specularity alone supports operational color constancy-the ability to discriminate changes in illumination or reflectance. Observers viewed short animations of illuminant or reflectance changes in rendered scenes containing a single spherical surface and were asked to classify the change. Performance improved with increasing specularity, as predicted from regularities in chromatic statistics. Peak performance was impaired by spatial rearrangements of image pixels that disrupted the perception of illuminated surfaces but was maintained with increased surface complexity. The characteristic chromatic transformations that are available with nonzero specularity are useful for operational color constancy, particularly if accompanied by appropriate perceptual organization.

Color constancy in a naturalistic, goal-directed task

In daily life, we use color information to select objects that will best serve a particular goal (e.g., pick the best-tasting fruit or avoid spoiled food). This is challenging when judgments must be made across changes in illumination as the spectrum reflected from an object to the eye varies with the illumination. Color constancy mechanisms serve to partially stabilize object color appearance across illumination changes, but whether and to what degree constancy supports accurate cross-illumination object selection is not well understood. To get closer to understanding how constancy operates in real-life tasks, we developed a paradigm in which subjects engage in a goal-directed task for which color is instrumental. Specifically, in each trial, subjects re-created an arrangement of colored blocks (the model) across a change in illumination. By analyzing the re-creations, we were able to infer and quantify the degree of color constancy that mediated subjects' performance. In Experiments 1 and 2, we used our paradigm to characterize constancy for two different sets of block reflectances, two different illuminant changes, and two different groups of subjects. On average, constancy was good in our naturalistic task, but it varied considerably across subjects. In Experiment 3, we tested whether varying scene complexity and the validity of local contrast as a cue to the illumination change modulated constancy. Increasing complexity did not lead to improved constancy; silencing local contrast significantly reduced constancy. Our results establish a novel goal-directed task that enables us to approach color constancy as it emerges in real life.

RenderToolbox3: MATLAB tools that facilitate physically based stimulus rendering for vision research

RenderToolbox3 provides MATLAB utilities and prescribes a workflow that should be useful to researchers who want to employ graphics in the study of vision and perhaps in other endeavors as well. In particular, RenderToolbox3 facilitates rendering scene families in which various scene attributes and renderer behaviors are manipulated parametrically, enables spectral specification of object reflectance and illuminant spectra, enables the use of physically based material specifications, helps validate renderer output, and converts renderer output to physical units of radiance. This paper describes the design and functionality of the toolbox and discusses several examples that demonstrate its use. We have designed RenderToolbox3 to be portable across computer hardware and operating systems and to be free and open source (except for MATLAB itself). RenderToolbox3 is available at https://github.com/DavidBrainard/RenderToolbox3.

The effect of background and illumination on color identification of real, 3D objects

For the surface reflectance of an object to be a useful cue to object identity, judgments of its color should remain stable across changes in the object's environment. In 2D scenes, there is general consensus that color judgments are much more stable across illumination changes than background changes. Here we investigate whether these findings generalize to real 3D objects. Observers made color matches to cubes as we independently varied both the illumination impinging on the cube and the 3D background of the cube. As in 2D scenes, we found relatively high but imperfect stability of color judgments under an illuminant shift. In contrast to 2D scenes, we found that background had little effect on average color judgments. In addition, variability of color judgments was increased by an illuminant shift and decreased by embedding the cube within a background. Taken together, these results suggest that in real 3D scenes with ample cues to object segregation, the addition of a background may improve stability of color identification.