Lightness identification of patterned three-dimensional, real objects

Perceiving the representative surface color of real-world materials

Natural surfaces such as soil, grass, and skin usually involve far more complex and heterogenous structures than the perfectly uniform surfaces assumed in studies on color and material perception. Despite this, we can easily perceive the representative color of these surfaces. Here, we investigated the visual mechanisms underlying the perception of representative surface color using 120 natural images of diverse materials and their statistically synthesized images. Our matching experiments indicated that the perceived representative color revealed was not significantly different from the Portilla-Simoncelli-synthesized images or phase-randomized images except for one sample, even though the perceived shape and material properties were greatly impaired in the synthetic stimuli. The results also showed that the matched representative colors were predictable from the saturation-enhanced color of the brightest point in the image, excluding the high-intensity outliers. The results support the notion that humans judge the representative color and lightness of real-world surfaces depending on simple image measurements.

Lightness Discrimination Depends More on Bright Rather Than Shaded Regions of Three-Dimensional Objects

The brighter portions of a shaded complex object are in principle more informative about its lightness and are preferentially fixated during lightness judgments. In this study, we investigate whether preventing this strategy also has measurable detrimental effects on performance. Observers were presented with a reference and a comparison three-dimensional rendered object and had to choose which one was "painted with a lighter gray." The comparison was rendered with different diffuse reflectance values. We compared precision between three different conditions: full image, 20% of the lightest pixels removed, or 20% of the darkest pixels removed. Removing the bright pixels maximally impaired performance. The results confirm that the strategy of relying on the brightest areas of a complex object in order to estimate lightness is functionally optimal, yielding more precise representations.

The perception of colour and material in naturalistic tasks

Perceived object colour and material help us to select and interact with objects. Because there is no simple mapping between the pattern of an object's image on the retina and its physical reflectance, our perceptions of colour and material are the result of sophisticated visual computations. A long-standing goal in vision science is to describe how these computations work, particularly as they act to stabilize perceived colour and material against variation in scene factors extrinsic to object surface properties, such as the illumination. If we take seriously the notion that perceived colour and material are useful because they help guide behaviour in natural tasks, then we need experiments that measure and models that describe how they are used in such tasks. To this end, we have developed selection-based methods and accompanying perceptual models for studying perceived object colour and material. This focused review highlights key aspects of our work. It includes a discussion of future directions and challenges, as well as an outline of a computational observer model that incorporates early, known, stages of visual processing and that clarifies how early vision shapes selection performance.

Why the "stimulus-error" did not go away

Psychologists in the early years of the discipline were much concerned with the stimulus-error. Roughly, this is the problem encountered in introspective experiments when subjects are liable to frame their perceptual reports in terms of what they know of the stimulus, instead of just drawing on their perceptual experiences as they are supposedly felt. "Introspectionist" psychologist E. B. Titchener and his student E. G. Boring both argued in the early 20th century that the stimulus-error is a serious methodological pit-fall. While many of the theoretical suppositions motivating Titchener and Boring have been unfashionable since the rise of behaviourism, the stimulus-error brings our attention to one matter of perennial importance to psychophysics and the psychology of perception. This is the fact that subjects are liable to give different kinds of perceptual reports in response to the same stimulus. I discuss attempts to control for variable reports in recent experimental work on colour and lightness constancy, and the disputes that have arisen over which kinds of reports are legitimate. Some contemporary psychologists do warn us against a stimulus-error, even though they do not use this terminology. I argue that concern over the stimulus-error is diagnostic of psychologists' deep theoretical commitments, such as their conception of sensation, or their demarcation of perception from cognition. I conclude by discussing the relevance of this debate to current philosophy of perception.

Transparent layer constancy under changes in illumination color: Does task matter?

If the perceived transmittances of transparent objects are matched across illuminants, only incomplete constancy is found. This implies that physically identical objects may appear different. Nevertheless, it was reported in the literature that subjects could almost perfectly identify such objects across illuminants, even if a transparency impression was suppressed by violating geometric transparency conditions. A potential interpretation of these findings is that (a) identification and matching rely on different criteria and processes and that (b) identification is solely based on low-level color information. We here present evidence against these hypotheses. Our results show that the best match, that is, the transparent object under the test illuminant that appears most similar to the standard, is also the preferred object in the identification task. Furthermore, we found that the degree of constancy increases in both tasks, if figural cues support a transparency impression and the accompanying color scission. We discuss the relation between matching and identification suggested by these findings.

Illumination frame of reference in the object-reviewing paradigm: A case of luminance and lightness

The present study combines the object-reviewing paradigm (Kahneman, Treisman, & Gibbs, 1992) with the checkershadow illusion (Adelson, 1995) to contrast the effects of objects' luminance versus lightness on the object-specific preview benefit. To this end, we manipulated objects' luminance and the amount of illumination given by an informative background scene in experiments. In line with previous studies (Moore, Stephens, & Hein, 2010), there was no object-specific preview benefit when objects were presented on a uniformly colored background and luminance switched between objects. In contrast, when objects were presented on the checkershadow illusion background which provided an explanation for the luminance switch, a reliable object-specific preview benefit was observed. This suggests that object correspondence as measured by the object-reviewing paradigm can be influenced by scene-induced, perceived lightness of objects' surfaces. We replicated this finding and moreover showed that the scene context only influences the object-specific preview benefit if the objects are perceived as part of the background scene.

Color constancy supports cross-illumination color selection

We rely on color to select objects as the targets of our actions (e.g., the freshest fish, the ripest fruit). To be useful for selection, color must provide accurate guidance about object identity across changes in illumination. Although the visual system partially stabilizes object color appearance across illumination changes, how such color constancy supports object selection is not understood. To study how constancy operates in real-life tasks, we developed a novel paradigm in which subjects selected which of two test objects presented under a test illumination appeared closer in color to a target object presented under a standard illumination. From subjects' choices, we inferred a selection-based match for the target via a variant of maximum likelihood difference scaling, and used it to quantify constancy. Selection-based constancy was good when measured using naturalistic stimuli, but was dramatically reduced when the stimuli were simplified, indicating that a naturalistic stimulus context is critical for good constancy. Overall, our results suggest that color supports accurate object selection across illumination changes when both stimuli and task match how color is used in real life. We compared our selection-based constancy results with data obtained using a classic asymmetric matching task and found that the adjustment-based matches predicted selection well for our stimuli and instructions, indicating that the appearance literature provides useful guidance for the emerging study of constancy in natural tasks.

Color constancy in a naturalistic, goal-directed task

In daily life, we use color information to select objects that will best serve a particular goal (e.g., pick the best-tasting fruit or avoid spoiled food). This is challenging when judgments must be made across changes in illumination as the spectrum reflected from an object to the eye varies with the illumination. Color constancy mechanisms serve to partially stabilize object color appearance across illumination changes, but whether and to what degree constancy supports accurate cross-illumination object selection is not well understood. To get closer to understanding how constancy operates in real-life tasks, we developed a paradigm in which subjects engage in a goal-directed task for which color is instrumental. Specifically, in each trial, subjects re-created an arrangement of colored blocks (the model) across a change in illumination. By analyzing the re-creations, we were able to infer and quantify the degree of color constancy that mediated subjects' performance. In Experiments 1 and 2, we used our paradigm to characterize constancy for two different sets of block reflectances, two different illuminant changes, and two different groups of subjects. On average, constancy was good in our naturalistic task, but it varied considerably across subjects. In Experiment 3, we tested whether varying scene complexity and the validity of local contrast as a cue to the illumination change modulated constancy. Increasing complexity did not lead to improved constancy; silencing local contrast significantly reduced constancy. Our results establish a novel goal-directed task that enables us to approach color constancy as it emerges in real life.

Accuracy and speed of material categorization in real-world images

It is easy to visually distinguish a ceramic knife from one made of steel, a leather jacket from one made of denim, and a plush toy from one made of plastic. Most studies of material appearance have focused on the estimation of specific material properties such as albedo or surface gloss, and as a consequence, almost nothing is known about how we recognize material categories like leather or plastic. We have studied judgments of high-level material categories with a diverse set of real-world photographs, and we have shown (Sharan, 2009) that observers can categorize materials reliably and quickly. Performance on our tasks cannot be explained by simple differences in color, surface shape, or texture. Nor can the results be explained by observers merely performing shape-based object recognition. Rather, we argue that fast and accurate material categorization is a distinct, basic ability of the visual system.

Selection of visual information for lightness judgements by eye movements

When judging the lightness of objects, the visual system has to take into account many factors such as shading, scene geometry, occlusions or transparency. The problem then is to estimate global lightness based on a number of local samples that differ in luminance. Here, we show that eye fixations play a prominent role in this selection process. We explored a special case of transparency for which the visual system separates surface reflectance from interfering conditions to generate a layered image representation. Eye movements were recorded while the observers matched the lightness of the layered stimulus. We found that observers did focus their fixations on the target layer, and this sampling strategy affected their lightness perception. The effect of image segmentation on perceived lightness was highly correlated with the fixation strategy and was strongly affected when we manipulated it using a gaze-contingent display. Finally, we disrupted the segmentation process showing that it causally drives the selection strategy. Selection through eye fixations can so serve as a simple heuristic to estimate the target reflectance.

Optimal sampling of visual information for lightness judgments

The variable resolution and limited processing capacity of the human visual system requires us to sample the world with eye movements and attentive processes. Here we show that where observers look can strongly modulate their reports of simple surface attributes, such as lightness. When observers matched the color of natural objects they based their judgments on the brightest parts of the objects; at the same time, they tended to fixate points with above-average luminance. When we forced participants to fixate a specific point on the object using a gaze-contingent display setup, the matched lightness was higher when observers fixated bright regions. This finding indicates a causal link between the luminance of the fixated region and the lightness match for the whole object. Simulations with rendered physical lighting show that higher values in an object's luminance distribution are particularly informative about reflectance. This sampling strategy is an efficient and simple heuristic for the visual system to achieve accurate and invariant judgments of lightness.

Visual inferences of material changes: color as clue and distraction

In a snapshot, a scene consists of things, but across time, the world consists of processes. Some are cyclical, for example, trees changing foliage through the seasons, surfaces getting wet and drying out; others are unidirectional, for example, fruit ripening and then decaying, or dust accumulating on surfaces. Chemical and physical properties of objects provide them with specific surface patterns of colors and textures. When endogenous and exogenous forces alter these colors and textures over time, the ability to identify these changes from appearances can have great utility in judging the composition, state, and history of objects. This short review presents thoughts on studying visual inferences of the properties of materials and their changes, including how to acquire calibrated images of time-varying materials, how to model time-varying appearance changes, how to measure observers' identification abilities, and how to parse out the perceptual qualities that help or hinder in recognizing materials and their states. For instance, if color information is removed, observers do significantly worse at recognizing materials and their changes, especially for organic materials. The role of color in object and scene recognition is still being debated, so elucidating color's role in material identification may also help to resolve the wider issue. This review introduces material change as an object of study in human perception and cognition, because the visual traces of changes are integral components of material and object identity. Visually based judgments of materials share the property of propensity with mental inferences, and conscious or unconscious visual imagery may play a role in setting expectancies for object shapes and properties. WIREs Cogni Sci 2011 2 686-700 DOI: 10.1002/wcs.148 This article is categorized under: Psychology > Perception and Psychophysics.

Color and material perception: achievements and challenges

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

Color strategies for object identification

We measured accuracy of object identification across illuminations on the basis of color cues. Four similarly shaped real objects, three of the same reflectance, were separated into pairs under distinct colored real lights. Observers were asked to pick the odd object. Correct and incorrect identifications formed systematic patterns that could not be explained by color-constancy, contrast-constancy, inverse-optics or neural-signal matching algorithms. The pattern of results were simulated by an algorithm that purposely made the incorrect assumption that color constancy holds, and used similarity between perceived object colors, along the difference vector between illuminant colors, to identify objects of the same reflectance across illuminants. The visual system may use this suboptimal strategy because the computational costs of an optimal strategy outweigh the benefits of more accurate performance.

Image statistics and the perception of surface qualities

The world is full of surfaces, and by looking at them we can judge their material qualities. Properties such as colour or glossiness can help us decide whether a pancake is cooked, or a patch of pavement is icy. Most studies of surface appearance have emphasized textureless matte surfaces, but real-world surfaces, which may have gloss and complex mesostructure, are now receiving increased attention. Their appearance results from a complex interplay of illumination, reflectance and surface geometry, which are difficult to tease apart given an image. If there were simple image statistics that were diagnostic of surface properties it would be sensible to use them. Here we show that the skewness of the luminance histogram and the skewness of sub-band filter outputs are correlated with surface gloss and inversely correlated with surface albedo (diffuse reflectance). We find evidence that human observers use skewness, or a similar measure of histogram asymmetry, in making judgements about surfaces. When the image of a surface has positively skewed statistics, it tends to appear darker and glossier than a similar surface with lower skewness, and this is true whether the skewness is inherent to the original image or is introduced by digital manipulation. We also find a visual after-effect based on skewness: adaptation to patterns with skewed statistics can alter the apparent lightness and glossiness of surfaces that are subsequently viewed. We suggest that there are neural mechanisms sensitive to skewed statistics, and that their outputs can be used in estimating surface properties.