Color and material perception: achievements and challenges

Conditions of a Multi-View 3D Display for Accurate Reproduction of Perceived Glossiness

Visualizing objects as they are perceived in the real world is often critical in our daily experiences. We previously focused on objects' surface glossiness visualized with a 3D display and found that a multi-view 3D display reproduces perceived glossiness more accurately than a 2D display. This improvement of glossiness reproduction can be explained by the fact that a glossy surface visualized by a multi-view 3D display appropriately provides luminance differences between the two eyes and luminance changes accompanying the viewer's lateral head motion. In the present study, to determine the requirements of a multi-view 3D display for the accurate reproduction of perceived glossiness, we developed a simulator of a multi-view 3D display to independently and simultaneously manipulate the viewpoint interval and the magnitude of the optical inter-view crosstalk. Using the simulator, we conducted a psychophysical experiment and found that glossiness reproduction is most accurate when the viewpoint interval is small and there is just a small (but not too small) amount of crosstalk. We proposed a simple yet perceptually valid model that quantitatively predicts the reproduction accuracy of perceived glossiness.

More "us," less "them": An appeal for pluralism - and stand-alone computational theorizing - in our science of social groups

The target article is an appeal to allow explicit computational theorizing into the study of social groups. Some commentators took this proposal and ran with it, some had questions about it, and some were confused or even put off by it. But even the latter did not seem to outright disagree - they thought the proposal was mutually exclusive with some other enterprise, when in fact it is not. Unfortunately, scientists studying social groups have not yet avoided the thread-bare trope of the blind men studying the different parts of the elephant: We see mutual exclusivity when we should see complementarity. I hope we can all take the next steps of examining how the different enterprises and approaches within our area of research might all fit together into a unified whole.

Quantification of Visual Texture and Presentation of Intermediate Visual Texture by Spatial Mixing

We proposed a method to display an intermediate visual texture by spatial mixing. In addition to color information, the visual texture is an important element that characterizes the nature of an object’s surface. While the system to display various color information has well matured in engineering, there is no method to reproduce visual textures in ambient light. In our method, the matte and glossy surfaces are used as “primary visual textures”, and an intermediate visual texture is displayed by spatially mixing the primary visual textures. In this paper, we first quantified the visual texture of an object's surface based on measured intensities of scattered and reflected lights. Next, based on the quantification, we evaluated spatially mixed surfaces consisting of two primary visual textures, an acrylic plate and a holed sheet of drawing paper, by changing the area proportion of the two primary visual textures. Finally, a sensory evaluation showed significant differences between each intermediate visual texture, and the results corresponded to a trend in the optical evaluation. This study illustrates that visual textures could be quantified based on the intensity of scattered and reflected light and reveals the applicability of our method to the display for intermediate visual texture.

Human brain activity reflecting facial attractiveness from skin reflection

Facial attraction has a great influence on our daily social interactions. Previous studies have mainly focused on the attraction from facial shape and expression. We recently found that faces with radiant skin appear to be more attractive than those with oily-shiny or matte skin. In the present study, we conducted functional magnetic resonance imaging (fMRI) and psychological experiments to determine the human brain activity that reflects facial attractiveness modulated by these skin reflection types. In the fMRI experiment, female subjects were shown successive images of unfamiliar female faces with matte, oily-shiny, or radiant skin. The subjects compared each face with the immediately preceding face in terms of attractiveness, age, and skin reflection, all based on the skin. The medial part of the orbitofrontal cortex (mOFC) was significantly more active when comparing attractiveness than when comparing skin reflection, suggesting that the mOFC is involved in processing facial attractiveness from skin reflection. In the psychological experiment, attractiveness rating was highest for radiant skin, followed by oily-shiny, and then matte skin. Comparison of the results of these experiments showed that mOFC activation level increased with attractiveness rating. These results suggest that the activation level of the mOFC reflects facial attractiveness from skin reflection.

The joint role of geometry and illumination on material recognition

Observing and recognizing materials is a fundamental part of our daily life. Under typical viewing conditions, we are capable of effortlessly identifying the objects that surround us and recognizing the materials they are made of. Nevertheless, understanding the underlying perceptual processes that take place to accurately discern the visual properties of an object is a long-standing problem. In this work, we perform a comprehensive and systematic analysis of how the interplay of geometry, illumination, and their spatial frequencies affects human performance on material recognition tasks. We carry out large-scale behavioral experiments where participants are asked to recognize different reference materials among a pool of candidate samples. In the different experiments, we carefully sample the information in the frequency domain of the stimuli. From our analysis, we find significant first-order interactions between the geometry and the illumination, of both the reference and the candidates. In addition, we observe that simple image statistics and higher-order image histograms do not correlate with human performance. Therefore, we perform a high-level comparison of highly nonlinear statistics by training a deep neural network on material recognition tasks. Our results show that such models can accurately classify materials, which suggests that they are capable of defining a meaningful representation of material appearance from labeled proximal image data. Last, we find preliminary evidence that these highly nonlinear models and humans may use similar high-level factors for material recognition tasks.

Expectations affect the perception of material properties

Many objects that we encounter have typical material qualities: spoons are hard, pillows are soft, and Jell-O dessert is wobbly. Over a lifetime of experiences, strong associations between an object and its typical material properties may be formed, and these associations not only include how glossy, rough, or pink an object is, but also how it behaves under force: we expect knocked over vases to shatter, popped bike tires to deflate, and gooey grilled cheese to hang between two slices of bread when pulled apart. Here we ask how such rich visual priors affect the visual perception of material qualities and present a particularly striking example of expectation violation. In a cue conflict design, we pair computer-rendered familiar objects with surprising material behaviors (a linen curtain shattering, a porcelain teacup wrinkling, etc.) and find that material qualities are not solely estimated from the object's kinematics (i.e., its physical [atypical] motion while shattering, wrinkling, wobbling etc.); rather, material appearance is sometimes “pulled” toward the “native” motion, shape, and optical properties that are associated with this object. Our results, in addition to patterns we find in response time data, suggest that visual priors about materials can set up high-level expectations about complex future states of an object and show how these priors modulate material appearance.

A Texture Selection Approach for Cultural Artifact 3D Reconstruction Considering Both Geometry and Radiation Quality

3D reconstruction of culture artifacts has great potential in digital heritage documentation and protection. Choosing the proper images for texture mapping from multi-view images is a major challenge for high precision and high quality 3D reconstruction of culture artifacts. In this study, a texture selection approach, considering both the geometry and radiation quality for 3D reconstruction of cultural artifacts while using multi-view dense matching is proposed. First, a Markov random field (MRF) method is presented to select images from the best angle of view among texture image sets. Then, an image radiation quality evaluation model is proposed in the virtue of a multiscale Tenengrad definition and brightness detection to eliminate fuzzy and overexposed textures. Finally, the selected textures are mapped to the 3D model under the mapping parameters of the multi-view dense matching and a semi-automatic texture mapping is executed on the 3DMax MudBox platform. Experimental results with two typical cultural artifacts data sets (bronze wares and porcelain) show that the proposed method can reduce abnormal exposure or fuzzy images to yield high quality 3D model of cultural artifacts.

Light: Toward a Transdisciplinary Science of Appearance and Atmosphere

To understand the processes behind seeing light, we need to integrate knowledge about the incoming optical structure, its perception, and how light interacts with material, shape, and space-objectively and subjectively. To that end, we need a novel approach to the science of light, namely, a transdisciplinary science of appearance, integrating optical, perceptual, and design knowledge and methods. In this article, I review existing literature as a basis for such a synthesis, which should discuss light in its full complexity, including its spatial properties and interactions with materials, shape, and space. I propose to investigate this by representing the endless variety of light, materials, shapes, and space as canonical modes and their combinations.

Perceptibility and Acceptability of Surface Gloss Variations in Dentistry

OBJECTIVE

To assess the visual perception of observers regarding perceptibility and acceptability of surface gloss variations of resin composites and to determine the limit of perceptibility and acceptability of gloss variations.

METHODS

Eight resin composite specimens and one human tooth specimen were fabricated. The resin specimens were polished to reach a surface gloss, in gloss units (GUs), of 10 GU, 20 GU, 30 GU, 40 GU, 50 GU, 60 GU, 70 GU, and 80 GU, and the human tooth specimen had a surface gloss of 80 GU. Sixty observers were selected to compare the surface gloss of the specimens in a light booth. For the perceptibility assessment, specimens were randomly displayed two at a time. Each observer performed a total of 144 observations. Observers answered two specific questions for determining the level and limit of perceptibility of gloss variations. The acceptability limit of gloss was determined by comparing the resin specimens with the tooth specimen. The observations were performed with dental practice scenarios (illuminant conditions, visualization field, and observers' education). Data were submitted to a nonlinear probit model and nonlinear regression estimation probit (5%).

RESULTS

Differences in perceptibility and acceptability were observed for surface gloss variations (ΔGU) (p<0.001). Perceptibility increased with ΔGU (10<20<30<40<50=60=70), while acceptability decreased with ΔGU (0=10>20>30>40>50>60=70). Acceptability and perceptibility limits were 6.4 GU and 35.7 GU, respectively.

CONCLUSIONS

Perceptibility and acceptability of surface gloss are influenced by gloss variations. A variation of 6.4 GU was required for 50% of observers to notice gloss variations. Concerning acceptability, observers required a variation of 35.7 GU to consider differences in gloss not acceptable.

Methods for Assessing Quantity and Quality of Illumination

Human vision provides useful information about the shape and color of the objects around us. It works well in many, but not all, lighting conditions. Since the advent of human-made light sources, it has been important to understand how illumination affects vision quality, but this has been surprisingly difficult. The widespread introduction of solid-state light emitters has increased the urgency of this problem. Experts still debate how lighting can best enable high-quality vision-a key issue since about one-fifth of global electrical power production is used to make light. Photometry, the measurement of the visual quantity of light, is well established, yet significant uncertainties remain. Colorimetry, the measurement of color, has achieved good reproducibility, but researchers still struggle to understand how illumination can best enable high-quality color vision. Fortunately, in recent years, considerable progress has been made. Here, we summarize the current understanding and discuss key areas for future study.

Neural Mechanisms of Material Perception: Quest on Shitsukan

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

Discrimination of spectral reflectance under environmental illumination

Color constancy is the ability to recover a stable perceptual estimate of surface reflectance, regardless of the lighting environment. However, we know little about how observers make judgments of the surface color of glossy objects, particularly in complex lighting environments that introduce complex spatial patterns of chromatic variation across an object's surface. To address this question, we measured thresholds for reflectance discrimination using computer-rendered stimuli under environmental illumination. In Experiment 1, we found that glossiness and shape had small effects on discrimination thresholds. Importantly, discrimination ellipses extended along the direction in which the chromaticities in the environmental illumination spread. In Experiment 2, we also found that the observers' abilities to judge surface colors were worse in lighting environments with an atypical chromatic distribution.

Perceptual gloss parameters are encoded by population responses in the monkey inferior temporal cortex

There are neurons localized in the lower bank of the superior temporal sulcus (STS) in the inferior temporal (IT) cortex of the monkey that selectively respond to specific ranges of gloss characterized by combinations of three physical reflectance parameters: specular reflectance (ρs), diffuse reflectance (ρd), and spread of specular reflection (α; Nishio et al., 2012). In the present study, we examined how the activities of these gloss-selective IT neurons are related to perceived gloss. In an earlier psychophysical study, Ferwerda et al. (2001) identified a perceptually uniform gloss space defined by two axes where the c-axis corresponds to a nonlinear combination of ρs and ρd and the d-axis corresponds to 1 - α. In the present study, we tested the responses of gloss-selective neurons to stimuli in the perceptual gloss space defined by the c- and d-axes. We found that gloss-selective neurons systematically changed their responses in the perceptual gloss space, and the distribution of the tuning directions of the population of gloss-selective neurons is biased toward directions in which perceived gloss increases. We also found that a set of perceptual gloss parameters as well as surface albedo can be well explained by the population activities of gloss-selective neurons, and that these parameters are likely encoded by the gloss-selective neurons in this area of the STS to represent various glosses. These results thus provide evidence that the IT cortex represents perceptual gloss space.

A Bayesian model of lightness perception that incorporates spatial variation in the illumination

The lightness of a test stimulus depends in a complex manner on the context in which it is viewed. To predict lightness, it is necessary to leverage measurements of a feasible number of contextual configurations into predictions for a wider range of configurations. Here we pursue this goal, using the idea that lightness results from the visual system's attempt to provide stable information about object surface reflectance. We develop a Bayesian algorithm that estimates both illumination and reflectance from image luminance, and link perceived lightness to the algorithm's estimates of surface reflectance. The algorithm resolves ambiguity in the image through the application of priors that specify what illumination and surface reflectances are likely to occur in viewed scenes. The prior distributions were chosen to allow spatial variation in both illumination and surface reflectance. To evaluate our model, we compared its predictions to a data set of judgments of perceived lightness of test patches embedded in achromatic checkerboards (Allred, Radonjić, Gilchrist, & Brainard, 2012). The checkerboard stimuli incorporated the large variation in luminance that is a pervasive feature of natural scenes. In addition, the luminance profile of the checks both near to and remote from the central test patches was systematically manipulated. The manipulations provided a simplified version of spatial variation in illumination. The model can account for effects of overall changes in image luminance and the dependence of such changes on spatial location as well as some but not all of the more detailed features of the data.

Neural selectivity and representation of gloss in the monkey inferior temporal cortex

When we view an object, its appearance depends in large part on specific surface reflectance properties; among these is surface gloss, which provides important information about the material composition of the object and the fine structure of its surface. To study how gloss is represented in the visual cortical areas related to object recognition, we examined the responses of neurons in the inferior temporal (IT) cortex of the macaque monkey to a set of object images exhibiting various combinations of specular reflection, diffuse reflection, and roughness, which are important physical parameters of surface gloss. We found that there are neurons in the lower bank of the superior temporal sulcus that selectively respond to specific gloss. This neuronal selectivity was largely maintained when the shape or illumination of the object was modified and perceived glossiness was unchanged. By contrast, neural responses were significantly altered when the pixels of the images were randomly rearranged, and perceived glossiness was dramatically changed. The stimulus preference of these neurons differed from cell to cell, and, as a population, they systematically represented a variety of surface glosses. We conclude that, within the visual cortex, there are mechanisms operating to integrate local image features and extract information about surface gloss and that this information is systematically represented in the IT cortex, an area playing an important role in object recognition.

Joint effects of illumination geometry and object shape in the perception of surface reflectance

Surface properties provide useful information for identifying objects and interacting with them. Effective utilization of this information, however, requires that the perception of object surface properties be relatively constant across changes in illumination and changes in object shape. Such constancy has been studied separately for changes in these factors. Here we ask whether the separate study of the illumination and shape effects is sufficient, by testing whether joint effects of illumination and shape changes can be predicted from the individual effects in a straightforward manner. We found large interactions between illumination and object shape in their effects on perceived glossiness. In addition, analysis of luminance histogram statistics could not account for the interactions.

Detection of changes in luminance distributions

How well can observers detect the presence of a change in luminance distributions? Performance was measured in three experiments. Observers viewed pairs of grayscale images on a calibrated CRT display. Each image was a checkerboard. All luminances in one image of each pair consisted of random draws from a single probability distribution. For the other image, some patch luminances consisted of random draws from that same distribution, while the rest of the patch luminances (test patches) consisted of random draws from a second distribution. The observers' task was to pick the image with luminances drawn from two distributions. The parameters of the second distribution that led to 75% correct performance were determined across manipulations of (1) the number of test patches, (2) the observers' certainty about test patch location, and (3) the geometric structure of the images. Performance improved with number of test patches and location certainty. The geometric manipulations did not affect performance. An ideal observer model with high efficiency fit the data well and a classification image analysis showed a similar use of information by the ideal and human observers, indicating that observers can make effective use of photometric information in our distribution discrimination task.

Surface color perception and equivalent illumination models

Vision provides information about the properties and identity of objects. The ease with which we perceive object properties belies the difficulty of the underlying information-processing task. In the case of object color, retinal information about object reflectance is confounded with information about the illumination as well as about the object's shape and pose. There is no obvious rule that allows transformation of the retinal image to a color representation that depends primarily on object surface reflectance. Under many circumstances, however, object color appearance is remarkably stable across scenes in which the object is viewed. Here, we review a line of experiments and theory that aim to understand how the visual system stabilizes object color appearance. Our emphasis is on models derived from explicit analysis of the computational problem of estimating the physical properties of illuminants and surfaces from the retinal image, and experiments that test these models. We argue that this approach has considerable promise for allowing generalization from simplified laboratory experiments to richer scenes that more closely approximate natural viewing. We discuss the relation between the work we review and other theoretical approaches available in the literature.