How direction of illumination affects visually perceived surface roughness

Material perception across different media-comparing perceived attributes in oil paintings and engravings

We investigated the influence of the medium on the perception of depicted objects and materials. Oil paintings and their reproductions in engravings were chosen because they are vastly distinctive media while having completely identical content. A total of 15 pairs were collected, consisting of 88 fragments depicting different materials, including fabric, skin, wood and metal. Besides the original condition, we created three manipulations to understand the effect of colour (a greyscale version) and contrast (equalised histograms towards both painting and engraving). We performed rating experiments on five attributes: three-dimensionality, glossiness, convincingness, smoothness and softness. An average of 25 participants finished each of the 20 online experimental sessions (five attributes X four conditions). Besides clear correlations between the two media, the differences mainly show in their means (different levels of perceived attributes) and standard deviations (perceived range). In most sessions, paintings depict a wider range than engravings. In addition, it was the histogram equalisation (global contrast) that made the most impact on perceived attributes, rather than colour removal. This suggests that engravers compensated for the lack of colour by exploiting the possibilities of local contrast.

Visual effects on tactile texture perception

How does vision affect active touch in judgments of surface roughness? We contrasted direct (combination of visual with tactile sensory information) and indirect (vision alters the processes of active touch) effects of vision on touch. Participants judged which of 2 surfaces was rougher using their index finger to make static contact with gratings of spatial period 1580 and 1620 μm. Simultaneously, they viewed the stimulus under one of five visual conditions: No vision, Filtered vision + touch, Veridical vision + touch (where vision alone yielded roughness discrimination at chance), Congruent vision + touch, Incongruent vision + touch. Results from 32 participants showed roughness discrimination for touch with vision was better than touch alone. The visual benefit for touch was strongest in a filtered (spatially non-informative) vision condition, thus results are interpreted in terms of indirect integration. An indirect effect of vision was further indicated by a finding of visual benefit in some but not all visuo-tactile congruency conditions.

Estimating lighting direction in scenes with multiple objects

To recover the reflectance and shape of an object in a scene, the human visual system must account for the properties of the light illuminating the object. Here, we examine the extent to which multiple objects within a scene are utilised to estimate the direction of lighting in a scene. In Experiment 1, we presented participants with rendered scenes that contained 1, 9, or 25 unfamiliar blob-like objects and measured their capacity to discriminate whether a directional light source was left or right of the participants' vantage point. Trends reported for ensemble perception suggest that the number of utilised objects-and, consequently, discrimination sensitivity-would increase with set size. However, we find little indication that increasing the number of objects in a scene increased discrimination sensitivity. In Experiment 2, an equivalent noise analysis was used to measure participants' internal noise and the number of objects used to judge the average light source direction in a scene, finding that participants relied on 1 or 2 objects to make their judgement regardless of whether 9 or 25 objects were present. In Experiment 3, participants completed a shape identification task that required an implicit judgement of light source direction, rather than an explicit judgement as in Experiments 1 and 2. We find that sensitivity for identifying surface shape was comparable for scenes containing 1, 9, and 25 objects. Our results suggest that the visual system relied on a small number of objects to estimate the direction of lighting in our rendered scenes.

Crossmodal associations between visual textures and temperature concepts

Visual textures are critical in how individuals form sensory expectations about objects, which include somatosensory properties such as temperature. This study aimed to uncover crossmodal associations between visual textures and temperature concepts. In Experiment 1 (N = 193), we evaluated crossmodal associations between 43 visual texture categories and different temperature concepts (via temperature words such as cold and hot) using an explicit forced-choice test. The results revealed associations between striped, cracked, matted, and waffled visual textures and high temperatures and between crystalline and flecked visual textures and low temperatures. In Experiment 2 (N = 247), we conducted six implicit association tests (IATs) pairing the two visual textures most strongly associated with low (crystalline and flecked) and high (striped and cracked) temperatures with the words cold and hot as per the results of Experiment 1. When pairing the crystalline and striped visual textures, the results revealed that crystalline was matched to the word cold, and striped was matched to the word hot. However, some associations found in the explicit test were not found in the IATs. In Experiment 3 (N = 124), we investigated how mappings between visual textures and concrete entities may influence crossmodal associations with temperature and these visual textures. Altogether, we found a range of association strengths and automaticity levels. Importantly, we found evidence of relative effects. Furthermore, some of these crossmodal associations are partly influenced by indirect mappings to concrete entities.

Experimentally disambiguating models of sensory cue integration

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behavior. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter's preferred framework. The aims of the present paper are to summarize and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Perception and Appreciation of Tactile Objects: The Role of Visual Experience and Texture Parameters

This exploratory study was designed to examine the effects of visual experience and specific texture parameters on both discriminative and aesthetic aspects of tactile perception. To this end, the authors conducted two experiments using a novel behavioral (ranking) approach in blind and (blindfolded) sighted individuals. Groups of congenitally blind, late blind, and (blindfolded) sighted participants made relative stimulus preference, aesthetic appreciation, and smoothness or softness judgment of two-dimensional (2D) or three-dimensional (3D) tactile surfaces through active touch. In both experiments, the aesthetic judgment was assessed on three affective dimensions, Relaxation, Hedonics, and Arousal, hypothesized to underlie visual aesthetics in a prior study. Results demonstrated that none of these behavioral judgments significantly varied as a function of visual experience in either experiment. However, irrespective of visual experience, significant differences were identified in all these behavioral judgments across the physical levels of smoothness or softness. In general, 2D smoothness or 3D softness discrimination was proportional to the level of physical smoothness or softness. Second, the smoother or softer tactile stimuli were preferred over the rougher or harder tactile stimuli. Third, the 3D affective structure of visual aesthetics appeared to be amodal and applicable to tactile aesthetics. However, analysis of the aesthetic profile across the affective dimensions revealed some striking differences between the forms of appreciation of smoothness and softness, uncovering unanticipated substructures in the nascent field of tactile aesthetics. While the physically softer 3D stimuli received higher ranks on all three affective dimensions, the physically smoother 2D stimuli received higher ranks on the Relaxation and Hedonics but lower ranks on the Arousal dimension. Moreover, the Relaxation and Hedonics ranks accurately overlapped with one another across all the physical levels of softness/hardness, but not across the physical levels of smoothness/roughness. These findings suggest that physical texture parameters not only affect basic tactile discrimination but differentially mediate tactile preferences, and aesthetic appreciation. The theoretical and practical implications of these novel findings are discussed.

Dynamic dot displays reveal material motion network in the human brain

There is growing research interest in the neural mechanisms underlying the recognition of material categories and properties. This research field, however, is relatively more recent and limited compared to investigations of the neural mechanisms underlying object and scene category recognition. Motion is particularly important for the perception of non-rigid materials, but the neural basis of non-rigid material motion remains unexplored. Using fMRI, we investigated which brain regions respond preferentially to material motion versus other types of motion. We introduce a new database of stimuli - dynamic dot materials - that are animations of moving dots that induce vivid percepts of various materials in motion, e.g. flapping cloth, liquid waves, wobbling jelly. Control stimuli were scrambled versions of these same animations and rigid three-dimensional rotating dots. Results showed that isolating material motion properties with dynamic dots (in contrast with other kinds of motion) activates a network of cortical regions in both ventral and dorsal visual pathways, including areas normally associated with the processing of surface properties and shape, and extending to somatosensory and premotor cortices. We suggest that such a widespread preference for material motion is due to strong associations between stimulus properties. For example viewing dots moving in a specific pattern not only elicits percepts of material motion; one perceives a flexible, non-rigid shape, identifies the object as a cloth flapping in the wind, infers the object's weight under gravity, and anticipates how it would feel to reach out and touch the material. These results are a first important step in mapping out the cortical architecture and dynamics in material-related motion processing.

Neural correlates of visual short-term memory for objects with material categories

Behavioral and neuroscience studies have shown that we can easily identify material categories, such as metal and fabric. Not only the early visual areas but also higher-order visual areas including the fusiform gyrus are known to be engaged in material perception. However, the brain mechanisms underlying visual short-term memory (VSTM) for material categories are unknown. To address this issue, we examined the neural correlates of VSTM for objects with material categories using a change detection task. In each trial, participants viewed a sample display containing two, four, or six objects having six material categories and were required to remember the locations and types of objects. After a brief delay, participants were asked to detect an object change based on the images or material categories in the test display (image-based and material-based conditions). Neuronal activity in the brain was assessed using functional magnetic resonance imaging (MRI). Behavioral results showed that the number of objects encoded did not increase as a function of set size in either image-based or material-based conditions. By contrast, MRI data showed a difference between the image-based and material-based conditions in percent signal change observed in a priori region of interest, the fusiform face area (FFA). Thus, we failed to achieve our research aim. However, the brain activation in the FFA correlated with the activation in the precentral/postcentral gyrus, which is related to haptic processing. Our findings indicate that the FFA may be involved in VSTM for objects with material categories in terms of the difference between images and material categories and that this memory may be mediated by the tactile properties of objects.

Light: Toward a Transdisciplinary Science of Appearance and Atmosphere

To understand the processes behind seeing light, we need to integrate knowledge about the incoming optical structure, its perception, and how light interacts with material, shape, and space-objectively and subjectively. To that end, we need a novel approach to the science of light, namely, a transdisciplinary science of appearance, integrating optical, perceptual, and design knowledge and methods. In this article, I review existing literature as a basis for such a synthesis, which should discuss light in its full complexity, including its spatial properties and interactions with materials, shape, and space. I propose to investigate this by representing the endless variety of light, materials, shapes, and space as canonical modes and their combinations.

Asymmetric perceptual confounds between canonical lightings and materials

To better understand the interactions between material perception and light perception, we further developed our material probe MatMix 1.0 into MixIM 1.0, which allows optical mixing of canonical lighting modes. We selected three canonical lighting modes (ambient, focus, and brilliance) and created scenes to represent the three illuminations. Together with four canonical material modes (matte, velvety, specular, glittery), this resulted in 12 basis images (the "bird set"). These images were optically mixed in our probing method. Three experiments were conducted with different groups of observers. In Experiment 1, observers were instructed to manipulate MixIM 1.0 and match optically mixed lighting modes while discounting the materials. In Experiment 2, observers were shown a pair of stimuli and instructed to simultaneously judge whether the materials and lightings were the same or different in a four-category discrimination task. In Experiment 3, observers performed both the matching and discrimination tasks in which only the ambient and focus light were implemented. Overall, the matching and discrimination results were comparable as (a) robust asymmetric perceptual confounds were found and confirmed in both types of tasks, (b) performances were consistent and all above chance levels, and (c) observers had higher sensitivities to our canonical materials than to our canonical lightings. The latter result may be explained in terms of a generic insensitivity for naturally occurring variations in light conditions. Our findings suggest that midlevel image features are more robust across different materials than across different lightings and, thus, more diagnostic for materials than for lightings, causing the asymmetric perceptual confounds.

Neural Mechanisms of Material Perception: Quest on Shitsukan

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

How accurately do different computer-based texture characterization methods predict material surface coarseness? A guideline for effective online inspection

The growing industrialization has emphasized the need for high-performance computer-based inspection methods. Here, we investigated the performance of four major computer-based texture characterization methods in the prediction of visually perceived and actual surface coarseness of real materials. Gray level co-occurrence matrix (GLCM), distance-dependent edge frequency (DDEF), fractal dimension (FD), and histogram skewness (SK) were used as the methods. A novel collection of real materials consisting of 20 sandpapers with high, medium, and low coarseness levels was employed. The results revealed that at high coarseness level the most precise prediction of actual surface coarseness was made by GLCM and SK, while in the prediction of visual coarseness all the methods worked similarly effectively. Perfect correlations were observed between GLCM, FD, and SK at visual and also actual coarseness at medium coarseness level. At low coarseness level, SK and DDEF acceptably predicted visual and actual coarseness, respectively. The image resolution impact on performance of the computer-based methods was found to be substantial. Results of the research present a guideline for choosing the best computer-based method as a viable substitute for the human observer in online inspections of materials' texture.

Visual Vibrometry: Estimating Material Properties from Small Motions in Video

The estimation of material properties is important for scene understanding, with many applications in vision, robotics, and structural engineering. This paper connects fundamentals of vibration mechanics with computer vision techniques in order to infer material properties from small, often imperceptible motions in video. Objects tend to vibrate in a set of preferred modes. The frequencies of these modes depend on the structure and material properties of an object. We show that by extracting these frequencies from video of a vibrating object, we can often make inferences about that object's material properties. We demonstrate our approach by estimating material properties for a variety of objects by observing their motion in high-speed and regular frame rate video.

Image Statistics and the Representation of Material Properties in the Visual Cortex

We explored perceived material properties (roughness, texturedness, and hardness) with a novel approach that compares perception, image statistics and brain activation, as measured with fMRI. We initially asked participants to rate 84 material images with respect to the above mentioned properties, and then scanned 15 of the participants with fMRI while they viewed the material images. The images were analyzed with a set of image statistics capturing their spatial frequency and texture properties. Linear classifiers were then applied to the image statistics as well as the voxel patterns of visually responsive voxels and early visual areas to discriminate between images with high and low perceptual ratings. Roughness and texturedness could be classified above chance level based on image statistics. Roughness and texturedness could also be classified based on the brain activation patterns in visual cortex, whereas hardness could not. Importantly, the agreement in classification based on image statistics and brain activation was also above chance level. Our results show that information about visual material properties is to a large degree contained in low-level image statistics, and that these image statistics are also partially reflected in brain activity patterns induced by the perception of material images.

Aesthetics by Numbers: Links between Perceived Texture Qualities and Computed Visual Texture Properties

Our world is filled with texture. For the human visual system, this is an important source of information for assessing environmental and material properties. Indeed-and presumably for this reason-the human visual system has regions dedicated to processing textures. Despite their abundance and apparent relevance, only recently the relationships between texture features and high-level judgments have captured the interest of mainstream science, despite long-standing indications for such relationships. In this study, we explore such relationships, as these might be used to predict perceived texture qualities. This is relevant, not only from a psychological/neuroscience perspective, but also for more applied fields such as design, architecture, and the visual arts. In two separate experiments, observers judged various qualities of visual textures such as beauty, roughness, naturalness, elegance, and complexity. Based on factor analysis, we find that in both experiments, ~75% of the variability in the judgments could be explained by a two-dimensional space, with axes that are closely aligned to the beauty and roughness judgments. That a two-dimensional judgment space suffices to capture most of the variability in the perceived texture qualities suggests that observers use a relatively limited set of internal scales on which to base various judgments, including aesthetic ones. Finally, for both of these judgments, we determined the relationship with a large number of texture features computed for each of the texture stimuli. We find that the presence of lower spatial frequencies, oblique orientations, higher intensity variation, higher saturation, and redness correlates with higher beauty ratings. Features that captured image intensity and uniformity correlated with roughness ratings. Therefore, a number of computational texture features are predictive of these judgments. This suggests that perceived texture qualities-including the aesthetic appreciation-are sufficiently universal to be predicted-with reasonable accuracy-based on the computed feature content of the textures.

On Visual Texture Preference: Can an Ecological Model Explain Why People Like Some Textures More Than Others?

What visual textures do people like and why? Here, we test whether the ecological valence theory proposed for color preferences can also predict people’s preferences for visual texture. According to the theory, people should like visual textures associated with positive objects or entities and dislike visual textures associated with negative objects or entities. We compare the results for the ecological model with a more traditional texture-preference model based on computational features and find that the ecological model performs reasonably well considering its lower complexity, explaining 63% of the variance in the human preference data.

Human visual cortical responses to specular and matte motion flows

Determining the compositional properties of surfaces in the environment is an important visual capacity. One such property is specular reflectance, which encompasses the range from matte to shiny surfaces. Visual estimation of specular reflectance can be informed by characteristic motion profiles; a surface with a specular reflectance that is difficult to determine while static can be confidently disambiguated when set in motion. Here, we used fMRI to trace the sensitivity of human visual cortex to such motion cues, both with and without photometric cues to specular reflectance. Participants viewed rotating blob-like objects that were rendered as images (photometric) or dots (kinematic) with either matte-consistent or shiny-consistent specular reflectance profiles. We were unable to identify any areas in low and mid-level human visual cortex that responded preferentially to surface specular reflectance from motion. However, univariate and multivariate analyses identified several visual areas; V1, V2, V3, V3A/B, and hMT+, capable of differentiating shiny from matte surface flows. These results indicate that the machinery for extracting kinematic cues is present in human visual cortex, but the areas involved in integrating such information with the photometric cues necessary for surface specular reflectance remain unclear.

Seeing liquids from static snapshots

Perceiving material properties can be crucial for many tasks-such as determining food edibility, or avoiding getting splashed-yet the visual perception of materials remains poorly understood. Most previous research has focussed on optical characteristics (e.g., gloss, translucency). Here, however, we show that shape also provides powerful visual cues to material properties. When liquids pour, splash or ooze, they organize themselves into characteristic shapes, which are highly diagnostic of the material's properties. Subjects viewed snapshots of simulated liquids of different viscosities, and rated their similarity. Using maximum likelihood difference scaling (Maloney & Yang, 2003), we reconstructed perceptual scales for perceived viscosity as a function of the physical viscosity of the simulated fluids. The resulting psychometric function revealed a distinct sigmoidal shape, distinguishing runny liquids that flow easily from viscous gels that clump up into piles. A parameter-free model based on 20 simple shape statistics predicted the subjects' data surprisingly well. This suggests that when subjects are asked to compare the viscosity of static snapshots of liquids that differ only in terms of viscosity, they rely primarily on relatively simple measures of shape similarity.

Invariant texture perception is harder with synthetic textures: Implications for models of texture processing

Texture synthesis models have become a popular tool for studying the representations supporting texture processing in human vision. In particular, the summary statistics implemented in the Portilla-Simoncelli (P-S) model support high-quality synthesis of natural textures, account for performance in crowding and search tasks, and may account for the response properties of V2 neurons. We chose to investigate whether or not these summary statistics are also sufficient to support texture discrimination in a task that required illumination invariance. Our observers performed a match-to-sample task using natural textures photographed with either diffuse overhead lighting or lighting from the side. Following a briefly presented sample texture, participants identified which of two test images depicted the same texture. In the illumination change condition, illumination differed between the sample and the matching test image. In the no change condition, sample textures and matching test images were identical. Critically, we generated synthetic versions of these images using the P-S model and also tested participants with these. If the statistics in the P-S model are sufficient for invariant texture perception, performance with synthetic images should not differ from performance in the original task. Instead, we found a significant cost of applying texture synthesis in both lighting conditions. We also observed this effect when power-spectra were matched across images (Experiment 2) and when sample and test images were drawn from unique locations in the parent textures to minimize the contribution of image-based processing (Experiment 3). Invariant texture processing thus depends upon measurements not implemented in the P-S algorithm.

The representation of material categories in the brain

Using textures mapped onto virtual nonsense objects, it has recently been shown that early visual cortex plays an important role in processing material properties. Here, we examined brain activation to photographs of materials, consisting of wood, stone, metal and fabric surfaces. These photographs were close-ups in the sense that the materials filled the image. In the first experiment, observers categorized the material in each image (i.e., wood, stone, metal, or fabric), while in an fMRI-scanner. We predicted the assigned material category using the obtained voxel patterns using a linear classifier. Region-of-interest and whole-brain analyses demonstrated material coding in the early visual regions, with lower accuracies for more anterior regions. There was little evidence for material coding in other brain regions. In the second experiment, we used an adaptation paradigm to reveal additional brain areas involved in the perception of material categories. Participants viewed images of wood, stone, metal, and fabric, presented in blocks with images of either different material categories (no adaptation) or images of different samples from the same material category (material adaptation). To measure baseline activation, blocks with the same material sample were presented (baseline adaptation). Material adaptation effects were found mainly in the parahippocampal gyrus, in agreement with fMRI-studies of texture perception. Our findings suggest that the parahippocampal gyrus, early visual cortex, and possibly the supramarginal gyrus are involved in the perception of material categories, but in different ways. The different outcomes from the two studies are likely due to inherent differences between the two paradigms. A third experiment suggested, based on anatomical overlap between activations, that spatial frequency information is important for within-category material discrimination.

Roughness perception of unfamiliar dot pattern textures

Both vision and touch yield comparable results in terms of roughness estimation of familiar textures as was shown in earlier studies. To our knowledge, no research has been conducted on the effect of sensory familiarity with the stimulus material on roughness estimation of unfamiliar textures. The influence of sensory modality and familiarity on roughness perception of dot pattern textures was investigated in a series of five experiments. Participants estimated the roughness of textures varying in mean center-to-center dot spacing in experimental conditions providing visual, haptic and visual-haptic combined information. The findings indicate that roughness perception of unfamiliar dot pattern textures is well described by a bi-exponential function of inter-dot spacing, regardless of the sensory modality used. However, sensory modality appears to affect the maximum of the psychophysical roughness function, with visually perceived roughness peaking for a smaller inter-dot spacing than haptic roughness. We propose that this might be due to the better spatial acuity of the visual modality. Individuals appeared to use different visual roughness estimation strategies depending on their first sensory experience (visual vs. haptic) with the stimulus material, primarily in an experimental context which required the combination of visual and haptic information in a single bimodal roughness estimate. Furthermore, the similarity of findings in experimental settings using real and virtual visual textures indicates the suitability of the experimental setup for neuroimaging studies, creating a more direct link between behavioral and neuroimaging results.

Visual and Haptic Representations of Material Properties

Research on material perception has received an increasing amount of attention recently. Clearly, both the visual and the haptic sense play important roles in the perception of materials, yet it is still unclear how both senses compare in material perception tasks. Here, we set out to investigate the degree of correspondence between the visual and the haptic representations of different materials. We asked participants to both categorize and rate 84 different materials for several material properties. In the haptic case, participants were blindfolded and asked to assess the materials based on haptic exploration. In the visual condition, participants assessed the stimuli based on their visual impressions only. While categorization performance was less consistent in the haptic condition than in the visual one, ratings correlated highly between the visual and the haptic modality. PCA revealed that all material samples were similarly organized within the perceptual space in both modalities. Moreover, in both senses the first two principal components were dominated by hardness and roughness. These are two material features that are fundamental for the haptic sense. We conclude that although the haptic sense seems to be crucial for material perception, the information it can gather alone might not be quite fine-grained and rich enough for perfect material recognition.

Visual motion and the perception of surface material

Many critical perceptual judgments, from telling whether fruit is ripe to determining whether the ground is slippery, involve estimating the material properties of surfaces. Very little is known about how the brain recognizes materials, even though the problem is likely as important for survival as navigating or recognizing objects. Though previous research has focused nearly exclusively on the properties of static images, recent evidence suggests that motion may affect the appearance of surface material. However, what kind of information motion conveys and how this information may be used by the brain is still unknown. Here, we identify three motion cues that the brain could rely on to distinguish between matte and shiny surfaces. We show that these motion measurements can override static cues, leading to dramatic changes in perceived material depending on the image motion characteristics. A classifier algorithm based on these cues correctly predicts both successes and some striking failures of human material perception. Together these results reveal a previously unknown use for optic flow in the perception of surface material properties.

Improving shape depiction under arbitrary rendering

Based on the observation that shading conveys shape information through intensity gradients, we present a new technique called Radiance Scaling that modifies the classical shading equations to offer versatile shape depiction functionalities. It works by scaling reflected light intensities depending on both surface curvature and material characteristics. As a result, diffuse shading or highlight variations become correlated with surface feature variations, enhancing concavities and convexities. The first advantage of such an approach is that it produces satisfying results with any kind of material for direct and global illumination: we demonstrate results obtained with Phong and Ashikmin-Shirley BRDFs, Cartoon shading, sub-Lambertian materials, perfectly reflective or refractive objects. Another advantage is that there is no restriction to the choice of lighting environment: it works with a single light, area lights, and interreflections. Third, it may be adapted to enhance surface shape through the use of precomputed radiance data such as Ambient Occlusion, Prefiltered Environment Maps or Lit Spheres. Finally, our approach works in real time on modern graphics hardware making it suitable for any interactive 3D visualization.

Color and material perception: achievements and challenges

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

The influence of shape cues on the perception of lighting direction

Three scene properties determine the luminances in the image of a shaded object: the material reflectance, the illuminant position, and the object's shape. Because all three properties determine the image, one cannot solve for any one property without knowing the other two. Nevertheless, people perceive consistent 3D shape and consistent lighting in shaded images; they must therefore be making assumptions about the unknown properties. We conducted two psychophysical experiments to determine how viewers use shape information to estimate the lighting direction from shaded images. In the first experiment, we confirmed that observers use 3D shape information when estimating lighting direction. In the second experiment, we investigated how different shape cues affect lighting direction estimates. Observers can accurately determine lighting direction when a host of shape cues specify the objects. When shading is the only cue, observers always set lighting direction to be from above. We modeled the results in a Bayesian framework that included a prior distribution describing the assumed lighting direction. The estimated prior was slightly counterclockwise from above at a ∼30° slant. Our model showed that an assumption of convexity provides an accurate estimate of lighting direction when the shape is globally, but not locally, consistent with convexity.

Detection of light transformations and concomitant changes in surface albedo

We report two experiments demonstrating that (1) observers are sensitive to information about changes in the light field not captured by local scene statistics and that (2) they can use this information to enhance detection of changes in surface albedo. Observers viewed scenes consisting of matte surfaces at many orientations illuminated by a collimated light source. All surfaces were achromatic, all lights neutral. In the first experiment, observers attempted to discriminate small changes in direction of the collimated light source (light transformations) from matched changes in the albedos of all surfaces (non-light transformations). Light changes and non-light changes shared the same local scene statistics and edge ratios, but the latter were not consistent with any change in direction to the collimated source. We found that observers could discriminate light changes as small as 5 degrees with sensitivity d' > 1 and accurately judge the direction of change. In a second experiment, we measured observers' ability to detect a change in the surface albedo of an isolated surface patch during either a light change or a surface change. Observers were more accurate in detecting isolated albedo changes during light changes. Measures of sensitivity d' were more than twice as great.

The role of visuohaptic experience in visually perceived depth

Berkeley suggested that "touch educates vision," that is, haptic input may be used to calibrate visual cues to improve visual estimation of properties of the world. Here, we test whether haptic input may be used to "miseducate" vision, causing observers to rely more heavily on misleading visual cues. Human subjects compared the depth of two cylindrical bumps illuminated by light sources located at different positions relative to the surface. As in previous work using judgments of surface roughness, we find that observers judge bumps to have greater depth when the light source is located eccentric to the surface normal (i.e., when shadows are more salient). Following several sessions of visual judgments of depth, subjects then underwent visuohaptic training in which haptic feedback was artificially correlated with the "pseudocue" of shadow size and artificially decorrelated with disparity and texture. Although there were large individual differences, almost all observers demonstrated integration of haptic cues during visuohaptic training. For some observers, subsequent visual judgments of bump depth were unaffected by the training. However, for 5 of 12 observers, training significantly increased the weight given to pseudocues, causing subsequent visual estimates of shape to be less veridical. We conclude that haptic information can be used to reweight visual cues, putting more weight on misleading pseudocues, even when more trustworthy visual cues are available in the scene.

Effect of pictorial depth cues, binocular disparity cues and motion parallax depth cues on lightness perception in three-dimensional virtual scenes

Background

Surface lightness perception is affected by scene interpretation. There is some experimental evidence that perceived lightness under bi-ocular viewing conditions is different from perceived lightness in actual scenes but there are also reports that viewing conditions have little or no effect on perceived color. We investigated how mixes of depth cues affect perception of lightness in three-dimensional rendered scenes containing strong gradients of illumination in depth.

Methodology/Principal Findings

Observers viewed a virtual room (4 m width×5 m height×17.5 m depth) with checkerboard walls and floor. In four conditions, the room was presented with or without binocular disparity (BD) depth cues and with or without motion parallax (MP) depth cues. In all conditions, observers were asked to adjust the luminance of a comparison surface to match the lightness of test surfaces placed at seven different depths (8.5–17.5 m) in the scene. We estimated lightness versus depth profiles in all four depth cue conditions. Even when observers had only pictorial depth cues (no MP, no BD), they partially but significantly discounted the illumination gradient in judging lightness. Adding either MP or BD led to significantly greater discounting and both cues together produced the greatest discounting. The effects of MP and BD were approximately additive. BD had greater influence at near distances than far.

Conclusions/Significance

These results suggest the surface lightness perception is modulated by three-dimensional perception/interpretation using pictorial, binocular-disparity, and motion-parallax cues additively. We propose a two-stage (2D and 3D) processing model for lightness perception.

Coding of images of materials by macaque inferior temporal cortical neurons

Objects vary not only in their shape but also in the material from which they are made. Knowledge of the material properties can contribute to object recognition as well as indicate properties of the object (e.g. ripeness of a fruit). We examined the coding of images of materials by single neurons of the macaque inferior temporal (IT) cortex, an area known to support object recognition and categorization. Stimuli were images of 12 real materials that were illuminated from three different directions. The material textures appeared within five different outline shapes. The majority of responsive IT neurons responded selectively to the material textures, and this selectivity was largely independent of their shape selectivity. The responses of the large majority of neurons were strongly affected by illumination direction. Despite the generally weak illumination-direction invariance of the responses, Support Vector Machines that used the neural responses as input were able to classify the materials across illumination direction better than by chance. A comparison between the responses to the original images and those to images with a random spectral phase, but matched power spectrum, indicated that the material texture selectivity did not depend merely on differences in the power spectrum but required phase information.

Conjoint measurement of gloss and surface texture

The image of a material's surface varies not only with viewing and illumination conditions, but also with the material's surface properties, including its 3-D texture and specularity. Previous studies on the visual perception of surface material have typically focused on single material properties, ignoring possible interactions. In this study, we used a conjoint-measurement design to determine how observers represent perceived 3-D texture ("bumpiness") and specularity ("glossiness") and modeled how each of these two surface-material properties affects perception of the other. Observers made judgments of bumpiness and glossiness of surfaces that varied in both surface texture and specularity. We quantified how changes in each surface-material property affected judgments of the other and found that a simple additive model captured visual perception of texture and specularity and their interaction. Conjoint measurement is potentially a powerful tool for analyzing perception of surface material in realistic environments.

Visual perception: a gloss on surface properties

How do humans interpret visual input to estimate the properties of a surface? In the case of estimation of gloss and lightness, it seems that neural discrimination of simple image statistics plays a large part.

The effect of viewpoint on perceived visual roughness

In previous work, we examined how the apparent roughness of a textured surface changed with direction of illumination. We found that observers exhibited systematic failures of roughness constancy across illumination conditions for triangular-faceted surfaces where physical roughness was defined as the variance of facet heights. These failures could be due, in part, to cues in the scene that confound changes in surface roughness with changes in illumination. These cues include the following: (1) the proportion of the surface in shadow, (2) mean luminance of the nonshadowed portion, (3) the standard deviation of the luminance of the nonshadowed portion, and (4) texture contrast. If the visual system relied on such "pseudocues" to roughness, then it would systematically misestimate surface roughness with changes in illumination much as our observers did despite the availability of depth cues such as binocular disparity. Here, we investigate observers' judgments of roughness when illumination direction and surface orientation are fixed and the observers' viewpoint with respect to the surface changes. We find a similar pattern of results. Observers exhibited patterned failures of roughness constancy with change in viewpoint, and an appreciable part of their failures could be accounted for by the same pseudocues. While the human visual system exhibits some degree of roughness constancy, our results lead to the conclusion that it does not always select the correct cues for a given visual task.