Perceptual transparency from image deformation

Unsupervised learning reveals interpretable latent representations for translucency perception

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is important but challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials, whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference from natural images. Here, we develop an unsupervised style-based image generation model to identify perceptually relevant dimensions for translucent material appearances from photographs. We find our model, with its layer-wise latent representation, can synthesize images of diverse and realistic materials. Importantly, without supervision, human-understandable scene attributes, including the object's shape, material, and body color, spontaneously emerge in the model's layer-wise latent space in a scale-specific manner. By embedding an image into the learned latent space, we can manipulate specific layers' latent code to modify the appearance of the object in the image. Specifically, we find that manipulation on the early-layers (coarse spatial scale) transforms the object's shape, while manipulation on the later-layers (fine spatial scale) modifies its body color. The middle-layers of the latent space selectively encode translucency features and manipulation of such layers coherently modifies the translucency appearance, without changing the object's shape or body color. Moreover, we find the middle-layers of the latent space can successfully predict human translucency ratings, suggesting that translucent impressions are established in mid-to-low spatial scale features. This layer-wise latent representation allows us to systematically discover perceptually relevant image features for human translucency perception. Together, our findings reveal that learning the scale-specific statistical structure of natural images might be crucial for humans to efficiently represent material properties across contexts.

Visual discrimination of optical material properties: A large-scale study

Complex visual processing involved in perceiving the object materials can be better elucidated by taking a variety of research approaches. Sharing stimulus and response data is an effective strategy to make the results of different studies directly comparable and can assist researchers with different backgrounds to jump into the field. Here, we constructed a database containing several sets of material images annotated with visual discrimination performance. We created the material images using physically based computer graphics techniques and conducted psychophysical experiments with them in both laboratory and crowdsourcing settings. The observer's task was to discriminate materials on one of six dimensions (gloss contrast, gloss distinctness of image, translucent vs. opaque, metal vs. plastic, metal vs. glass, and glossy vs. painted). The illumination consistency and object geometry were also varied. We used a nonverbal procedure (an oddity task) applicable for diverse use cases, such as cross-cultural, cross-species, clinical, or developmental studies. Results showed that the material discrimination depended on the illuminations and geometries and that the ability to discriminate the spatial consistency of specular highlights in glossiness perception showed larger individual differences than in other tasks. In addition, analysis of visual features showed that the parameters of higher order color texture statistics can partially, but not completely, explain task performance. The results obtained through crowdsourcing were highly correlated with those obtained in the laboratory, suggesting that our database can be used even when the experimental conditions are not strictly controlled in the laboratory. Several projects using our dataset are underway.

The look and feel of soft are similar across different softness dimensions

The softness of objects can be perceived through several senses. For instance, to judge the softness of a cat's fur, we do not only look at it, we often also run our fingers through its coat. Recently, we have shown that haptically perceived softness covaries with the compliance, viscosity, granularity, and furriness of materials (Dovencioglu, Üstün, Doerschner, & Drewing, 2020). However, it is unknown whether vision can provide similar information about the various aspects of perceived softness. Here, we investigated this question in an experiment with three conditions: in the haptic condition, blindfolded participants explored materials with their hands, in the static visual condition participants were presented with close-up photographs of the same materials, and in the dynamic visual condition participants watched videos of the hand-material interactions that were recorded in the haptic condition. After haptically or visually exploring the materials, participants rated them on various attributes. Our results show a high overall perceptual correspondence among the three experimental conditions. With a few exceptions, this correspondence tended to be strongest between haptic and dynamic visual conditions. These results are discussed with respect to information potentially available through the senses, or through prior experience, when judging the softness of materials.

Recognising the dynamic form of fire

Encoding and recognising complex natural sequences provides a challenge for human vision. We found that observers could recognise a previously presented segment of a video of a hearth fire when embedded in a longer sequence. Recognition performance declined when the test video was spatially inverted, but not when it was hue reversed or temporally reversed. Sampled motion degraded forwards/reversed playback discrimination, indicating observers were sensitive to the asymmetric pattern of motion of flames. For brief targets, performance increased with target length. More generally, performance depended on the relative lengths of the target and embedding sequence. Increased errors with embedded sequence length were driven by positive responses to non-target sequences (false alarms) rather than omissions. Taken together these observations favour interpreting performance in terms of an incremental decision-making model based on a sequential statistical analysis in which evidence accrues for one of two alternatives. We also suggest that prediction could provide a means of providing and evaluating evidence in a sequential analysis model.

Softness and weight from shape: Material properties inferred from local shape features

Object shape is an important cue to material identity and for the estimation of material properties. Shape features can affect material perception at different levels: at a microscale (surface roughness), mesoscale (textures and local object shape), or megascale (global object shape) level. Examples for local shape features include ripples in drapery, clots in viscous liquids, or spiraling creases in twisted objects. Here, we set out to test the role of such shape features on judgments of material properties softness and weight. For this, we created a large number of novel stimuli with varying surface shape features. We show that those features have distinct effects on softness and weight ratings depending on their type, as well as amplitude and frequency, for example, increasing numbers and pointedness of spikes makes objects appear harder and heavier. By also asking participants to name familiar objects, materials, and transformations they associate with our stimuli, we can show that softness and weight judgments do not merely follow from semantic associations between particular stimuli and real-world object shapes. Rather, softness and weight are estimated from surface shape, presumably based on learned heuristics about the relationship between a particular expression of surface features and material properties. In line with this, we show that correlations between perceived softness or weight and surface curvature vary depending on the type of surface feature. We conclude that local shape features have to be considered when testing the effects of shape on the perception of material properties such as softness and weight.

A Computational Mechanism for Seeing Dynamic Deformation

Human observers perceptually discriminate the dynamic deformation of materials in the real world. However, the psychophysical and neural mechanisms responsible for the perception of dynamic deformation have not been fully elucidated. By using a deforming bar as the stimulus, we showed that the spatial frequency of deformation was a critical determinant of deformation perception. Simulating the response of direction-selective units (i.e., MT pattern motion cells) to stimuli, we found that the perception of dynamic deformation was well explained by assuming a higher-order mechanism monitoring the spatial pattern of direction responses. Our model with the higher-order mechanism also successfully explained the appearance of a visual illusion wherein a static bar apparently deforms against a tilted drifting grating. In particular, it was the lower spatial frequencies in this pattern that strongly contributed to the deformation perception. Finally, by manipulating the luminance of the static bar, we observed that the mechanism for the illusory deformation was more sensitive to luminance than contrast cues.

Mid-Air Action Contributes to Pseudo-Haptic Stiffness Effects

Pseudo-haptic feedback takes advantage of a cross-modal integration between vision and haptics. Previous studies have shown that object stiffness can be rendered with pseudo-haptic feedback with external haptic inputs. This article explored whether the pseudo-haptic feedback was feasible with a mid-air action wherein no external haptic input was given. On each trial of the experiments, participants conduced a mid-air action to laterally move their hands as if they horizontally stretched an object in the display. In synchronized with the hands' motion, the object horizontally deformed. The magnitude of the object deformation varied with the horizontal distance between participants' hands (i.e., a hand distance). The ratio of deformation magnitudes to the hand distance (i.e., a deformation-to-distance ratio) was controlled; With a larger ratio, a smaller hand distance produced the maximum level of object deformation. The Poisson's ratio was also controlled; a higher Poisson's ratio produced a larger magnitude of vertical deformation. The participants were asked to report the stiffness of the objects with a five-point rating scale. Consequently, the stiffness rating decreased with the deformation-distance ratio and with the Poisson's ratio. The results indicate that pseudo-haptic stiffness can be rendered with mid-air action by manipulating the deformation-distance ratio and Poisson's ratio.

Visual shape perception in the case of transparent objects

In order to estimate the shape of objects, the visual system must refer to shape-related regularities in the (retinal) image. For opaque objects, many such regularities have already been identified, but most of them cannot simply be transferred to transparent objects, because they are not available there at all or are available only in a substantially modified form. We here consider three potentially relevant regularities specific to transparent objects: optical background distortions due to refraction, changes in chromaticity and brightness due to absorption, and multiple mirror images due to specular reflection. Using computer simulations, we first analyze under which conditions these regularities may be used as shape cues. We further investigate experimentally how shape perception depends on the availability of these potential cues in realistic scenes under natural viewing conditions. Our results show that the shape of transparent objects was perceived both less accurately and less precisely than in the opaque case. Furthermore, the influence of individual image regularities varied considerably depending on the properties of both object and scene. This suggests that in the transparent case, what kind of information is usable as a shape cue depends on a complex interplay of properties of the transparent object and the surrounding scene.

Perceptual Transparency From Cast Shadow

This study examined how a shadow contributes to the perception of a transparent surface. As stimuli, we used computer graphics images in which a transparent surface with a color-mosaic pattern casts a shadow onto a background surface. We manipulated two parameters: (a) the spatial heterogeneity of the transmittance of the transparent surface and (b) the size of the light source shining on the transparent surface and its background. The latter parameter determined the blurriness of shadows. Observers judged whether the stimulus image contained a transparent surface or not. We found that the proportion of reports identifying a transparent surface was dependent on both parameters we tested. Specifically, a high spatial heterogeneity of transmittance decreased the proportion of reports of a transparent surface; this was possibly because globally defined X-junctions, which were one of the cues to perceptual transparency, perceptually broke down. On the other hand, blurred shadows were effective even when the global X-junctions were not effective. Locally defined X-junctions only moderately contributed to perceptual transparency. The results indicate that in addition to global and local X-junctions, blurred shadows are image features that elicit the perception of transparency from a cast shadow. A large individual difference as to which information each participant used as a cue to perceptual transparency was also discussed.

Seeing through transparent layers

The human visual system is remarkably good at decomposing local and global deformations in the flow of visual information into different perceptual layers, a critical ability for daily tasks such as driving through rain or fog or catching that evasive trout. In these scenarios, changes in the visual information might be due to a deforming object or deformations due to a transparent medium, such as structured glass or water, or a combination of these. How does the visual system use image deformations to make sense of layering due to transparent materials? We used eidolons to investigate equivalence classes for perceptually similar transparent layers. We created a stimulus space for perceptual equivalents of a fiducial scene by systematically varying the local disarray parameters reach and grain. This disarray in eidolon space leads to distinct impressions of transparency, specifically, high reach and grain values vividly resemble water whereas smaller grain values appear diffuse like structured glass. We asked observers to adjust image deformations so that the objects in the scene looked like they were seen (a) under water, (b) behind haze, or (c) behind structured glass. Observers adjusted image deformation parameters by moving the mouse horizontally (grain) and vertically (reach). For two conditions, water and glass, we observed high intraobserver consistency: responses were not random. Responses yielded a concentrated equivalence class for water and structured glass.

Linear Motion Coverage as a Determinant of Transparent Liquid Perception

When a transparent liquid flows, the background image behind the flow dynamically deforms due to light refraction. The dynamic deformations of a background image (dynamic image deformations) are one of the visual features used by the visual system to infer the existence of a transparent liquid flow. Although previous studies have discussed the role of the narrow band components of the spatiotemporal deformation frequency, it was still unclear whether motion signals, one of the constituents of dynamic image deformations, were the determinant of the perception of a transparent liquid. Manipulating the flow speed of image deformation, which is a critical parameter for changing motion signals in dynamic image deformations, we asked observers to judge whether a transparent liquid was included in the clips or not. We found that the proportions of reporting that they saw a transparent liquid increased with the flow speed of image deformations. Analyzing motion signals of the stimulus clips, we found that the faster the flow of image deformations the fewer linear motion signals were contained. The results indicate that the perception of a transparent liquid arises when the dynamic image deformations contain fewer linear motion signals.

Neural Mechanisms of Material Perception: Quest on Shitsukan

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

Deformation-induced transparency resolves color scission

Dynamic image deformation produces the perception of a transparent material that appears to deform the background image by light refraction. Since past studies on this phenomenon have mainly used subjective judgment about the presence of a transparent layer, it remains unsolved whether this is a real perceptual transparency effect in the sense that it forms surface representations, as do conventional transparency effects. Visual computation for color and luminance transparency, induced mainly by surface-contour information, can be decomposed into two components: surface formation to determine foreground and background layers, and scission to assign color and luminance to each layer. Here we show that deformation-induced perceptual transparency aids surface formation by color transparency and consequently resolves color scission. We asked observers to report the color of the front layer in a spatial region with a neutral physical color. The layer color could be seen as either reddish or greenish depending on the spatial context producing the color transparency, which was, however, ambiguous about the order of layers. We found that adding to the display a deformation-induced transparency that could specify the front layer significantly biased color scission in the predicted way if and only if the deformation-induced transparency was spatially coincident with the interpretation of color transparency. The results indicate that deformation-induced transparency is indeed a novel type of perceptual transparency that plays a role in surface formation in cooperation with color transparency.

Motion Perception: From Detection to Interpretation

Visual motion processing can be conceptually divided into two levels. In the lower level, local motion signals are detected by spatiotemporal-frequency-selective sensors and then integrated into a motion vector flow. Although the model based on V1-MT physiology provides a good computational framework for this level of processing, it needs to be updated to fully explain psychophysical findings about motion perception, including complex motion signal interactions in the spatiotemporal-frequency and space domains. In the higher level, the velocity map is interpreted. Although there are many motion interpretation processes, we highlight the recent progress in research on the perception of material (e.g., specular reflection, liquid viscosity) and on animacy perception. We then consider possible linking mechanisms of the two levels and propose intrinsic flow decomposition as the key problem. To provide insights into computational mechanisms of motion perception, in addition to psychophysics and neurosciences, we review machine vision studies seeking to solve similar problems.

Dynamic Visual Cues for Differentiating Mirror and Glass

Mirror materials (perfect specular surfaces such as polished metal) and glass materials (transparent and refraction media) are quite commonly encountered in everyday life. The human visual system can discriminate these complex distorted images formed by reflection or transmission of the surrounding environment even though they do not intrinsically possess surface colour. In this study, we determined the cues that aid mirror and glass discrimination. From video analysis, we found that glass objects have more opposite motion components relative to the direction of object rotation. Then, we hypothesised a model developed using motion transparency because motion information is not only present on the front side, but also on the rear side of the object surface in the glass material object. In materials judging experiments, we found that human performance with rotating video stimuli is higher than that with static stimuli (simple images). Subsequently, we compared the developed model derived from motion coherency to human rating performance for transparency and specular reflection. The model sufficiently identified the different materials using dynamic information. These results suggest that the visual system relies on dynamic cues that indicate the difference between mirror and glass.

Interactive Dynamic Volume Illumination with Refraction and Caustics

In recent years, significant progress has been made in developing high-quality interactive methods for realistic volume illumination. However, refraction - despite being an important aspect of light propagation in participating media - has so far only received little attention. In this paper, we present a novel approach for refractive volume illumination including caustics capable of interactive frame rates. By interleaving light and viewing ray propagation, our technique avoids memory-intensive storage of illumination information and does not require any precomputation. It is fully dynamic and all parameters such as light position and transfer function can be modified interactively without a performance penalty.

Visual discomfort and flicker

Flickering lights can be uncomfortable to look at and can induce seizures in observers with photosensitive epilepsy. However, the temporal characteristics contributing to these effects are not fully known. In the spatial domain, one identified source of visual discomfort is when images have Fourier amplitude spectra that deviate from the natural (∼1/frequency, 1/f) statistical characteristics of natural scenes, especially if they contain excess energy at the medium frequencies at which the visual system is most sensitive. We tested for analogous effects in the temporal domain, manipulating both the amplitude and phase spectra of the flicker. Participants judged the relative discomfort of temporal luminance variations in a pair of uniform 17° fields with different temporal modulations. In general, discomfort increased with deviations from natural amplitude spectra, particularly those with excess energy at medium frequencies or biased toward sharper spectra. These ratings of discomfort were also consistent with ratings of how natural the modulations appeared. However, the temporal discomfort judgments were also strongly affected by the phase spectra of the flicker, with fixed vs. random spectra producing very different responses. This was not due to the perceived regularity or predictability of the flicker, but could arise from a number of other potential factors. Our findings suggest that, like spatial patterns, visual discomfort in time-varying patterns depends in part on how similar they are to the amplitude spectra of temporal variations in the natural visual environment, but also point to the critical role of the phase spectrum in the perceived discomfort of flicker.

What Property of the Contour of a Deforming Region Biases Percepts toward Liquid?

Human observers can perceive the existence of a transparent surface from dynamic image deformation. They can also easily discriminate a transparent solid material such as plastic and glass from a transparent fluid one such as water and shampoo just by viewing them. However, the image information required for material discrimination of this sort is still unclear. A liquid changes its contour shape non-rigidly. We therefore examined whether additional properties of the contour of a deformation-defined region, which indicated contour non-rigidity, biased percepts of the region toward liquid materials. Our stimuli had a translating circular region wherein a natural texture image was deformed at the spatiotemporal deformation frequency that was optimal for the perception of a transparent layer. In Experiment 1, we dynamically deformed the contour of the circular region and found that large deformation of the contour biased the percept toward liquid. In Experiment 2, we manipulated the blurriness of the contour and observed that a strongly blurred contour biased percepts toward liquid. Taken together, the results suggest that a deforming region lacking a discrete contour biases percepts toward liquid.

Image deformation as a cue to material category judgment

Human observers easily recognize complex natural phenomena, such as flowing water, which often generate highly chaotic dynamic arrays of light on the retina. It has not been clarified how the visual system discerns the source of a fluid flow. Here we show that the magnitude of image deformation caused by light refraction is a critical factor for the visual system to determine the perceptual category of fluid flows. Employing a physics engine, we created computer-rendered scenes of water and hot air flows. For each flow, we manipulated the rendering parameters (distortion factors and the index of refraction) that strongly influence the magnitude of image deformation. The observers rated how strongly they felt impressions of water and hot air in the video clips of the flows. The ratings showed that the water and hot air impressions were positively and negatively related to the magnitude of image deformation. Based on the results, we discuss how the visual system heuristically utilizes image deformation to discern non-rigid materials such as water and hot air flows.