Neural Mechanisms of Material Perception: Quest on Shitsukan

Glossiness perception and its pupillary response

Recent studies have revealed that pupillary response changes depend on perceptual factors such as subjective brightness caused by optical illusions and luminance. However, the manner in which the perceptual factor that is derived from the glossiness perception of object surfaces affects the pupillary response remains unclear. We investigated the relationship between the glossiness perception and pupillary response through a glossiness rating experiment that included recording the pupil diameter. We prepared general object images (original) and randomized images (shuffled) that comprised the same images with randomized small square regions as stimuli. The image features were controlled by matching the luminance histogram. The observers were asked to rate the perceived glossiness of the stimuli presented for 3,000 ms and the changes in their pupil diameters were recorded. Images with higher glossiness ratings constricted the pupil size more than those with lower glossiness ratings at the peak constriction of the pupillary responses during the stimulus duration. The linear mixed-effects model demonstrated that the glossiness rating, image category (original/shuffled), variance of the luminance histogram, and stimulus area were most effective in predicting the pupillary responses. These results suggest that the illusory brightness obtained by the image regions of high-glossiness objects, such as specular highlights, induce pupil constriction.

Probing the Link Between Vision and Language in Material Perception Using Psychophysics and Unsupervised Learning

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to express our subjective understanding of visual input and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression in material perception to understand how visual features relate to semantic representations. We use deep generative networks to construct an expandable image space to systematically create materials of well-defined and ambiguous categories. From such a space, we sampled diverse stimuli and compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual features that characterize the material's optical properties. Analyzing the image representation of materials obtained from various pre-trained data-rich deep neural networks, we find that human visual judgments' similarity structures align more closely with those of the text-guided visual-semantic model than purely vision-based models. Our findings suggest that while semantic representations facilitate material categorization, non-semantic visual features also play a significant role in discriminating materials at a finer level. This work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for quantitatively evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

materialmodifier: An R package of photo editing effects for material perception research

In this paper, we introduce an R package that performs automated photo editing effects. Specifically, it is an R implementation of an image-processing algorithm proposed by Boyadzhiev et al. (2015). The software allows the user to manipulate the appearance of objects in photographs, such as emphasizing facial blemishes and wrinkles, smoothing the skin, or enhancing the gloss of fruit. It provides a reproducible method to quantitatively control specific surface properties of objects (e.g., gloss and roughness), which is useful for researchers interested in topics related to material perception, from basic mechanisms of perception to the aesthetic evaluation of faces and objects. We describe the functionality, usage, and algorithm of the method, report on the findings of a behavioral evaluation experiment, and discuss its usefulness and limitations for psychological research. The package can be installed via CRAN, and documentation and source code are available at https://github.com/tsuda16k/materialmodifier .

Sustainable materials: a linking bridge between material perception, affordance, and aesthetics

The perception of material properties, which refers to the way in which individuals perceive and interpret materials through their sensory experiences, plays a crucial role in our interaction with the environment. Affordance, on the other hand, refers to the potential actions and uses that materials offer to users. In turn, the perception of the affordances is modulated by the aesthetic appreciation that individuals experience when interacting with the environment. Although material perception, affordances, and aesthetic appreciation are recognized as essential to fostering sustainability in society, only a few studies have investigated this subject matter systematically and their reciprocal influences. This scarcity is partially due to the challenges offered by the complexity of combining interdisciplinary topics that explore interactions between various disciplines, such as psychophysics, neurophysiology, affective science, aesthetics, and social and environmental sciences. Outlining the main findings across disciplines, this review highlights the pivotal role of material perception in shaping sustainable behaviors. It establishes connections between material perception, affordance, aesthetics, and sustainability, emphasizing the need for interdisciplinary research and integrated approaches in environmental psychology. This integration is essential as it can provide insight into how to foster sustainable and durable changes.

Second-order boundaries segment more easily when they are density-defined rather than feature-defined

Previous studies have demonstrated that density is an important perceptual aspect of textural appearance to which the visual system is highly attuned. Furthermore, it is known that density cues not only influence texture segmentation, but can enable segmentation by themselves, in the absence of other cues. A popular computational model of texture segmentation known as the "Filter-Rectify-Filter" (FRF) model predicts that density should be a second-order cue enabling segmentation. For a compound texture boundary defined by superimposing two single-micropattern density boundaries, a version of the FRF model in which different micropattern-specific channels are analyzed separately by different second-stage filters makes the prediction that segmentation thresholds should be identical in two cases: (1) Compound boundaries with an equal number of micropatterns on each side but different relative proportions of each variety (compound feature boundaries) and (2) Compound boundaries with different numbers of micropatterns on each side, but with each side having an identical number of each variety (compound density boundaries). We directly tested this prediction by comparing segmentation thresholds for second-order compound feature and density boundaries, comprised of two superimposed single-micropattern density boundaries comprised of complementary micropattern pairs differing either in orientation or contrast polarity. In both cases, we observed lower segmentation thresholds for compound density boundaries than compound feature boundaries, with identical results when the compound density boundaries were equated for RMS contrast. In a second experiment, we considered how two varieties of micropatterns summate for compound boundary segmentation. In the case where two single micro-pattern density boundaries are superimposed to form a compound density boundary, we find that the two channels combine via probability summation. By contrast, when they are superimposed to form a compound feature boundary, segmentation performance is worse than for either channel alone. From these findings, we conclude that density segmentation may rely on neural mechanisms different from those which underlie feature segmentation, consistent with recent findings suggesting that density comprises a separate psychophysical 'channel'.

Analysis of convolutional neural networks reveals the computational properties essential for subcortical processing of facial expression

Perception of facial expression is crucial for primate social interactions. This visual information is processed through the ventral cortical pathway and the subcortical pathway. However, the subcortical pathway exhibits inaccurate processing, and the responsible architectural and physiological properties remain unclear. To investigate this, we constructed and examined convolutional neural networks with three key properties of the subcortical pathway: a shallow layer architecture, concentric receptive fields at the initial processing stage, and a greater degree of spatial pooling. These neural networks achieved modest accuracy in classifying facial expressions. By replacing these properties, individually or in combination, with corresponding cortical features, performance gradually improved. Similar to amygdala neurons, some units in the final processing layer exhibited sensitivity to retina-based spatial frequencies (SFs), while others were sensitive to object-based SFs. Replacement of any of these properties affected the coordinates of the SF encoding. Therefore, all three properties limit the accuracy of facial expression information and are essential for determining the SF representation coordinate. These findings characterize the role of the subcortical computational processes in facial expression recognition.

Material category of visual objects computed from specular image structure

Recognizing materials and their properties visually is vital for successful interactions with our environment, from avoiding slippery floors to handling fragile objects. Yet there is no simple mapping of retinal image intensities to physical properties. Here, we investigated what image information drives material perception by collecting human psychophysical judgements about complex glossy objects. Variations in specular image structure-produced either by manipulating reflectance properties or visual features directly-caused categorical shifts in material appearance, suggesting that specular reflections provide diagnostic information about a wide range of material classes. Perceived material category appeared to mediate cues for surface gloss, providing evidence against a purely feedforward view of neural processing. Our results suggest that the image structure that triggers our perception of surface gloss plays a direct role in visual categorization, and that the perception and neural processing of stimulus properties should be studied in the context of recognition, not in isolation.

Crossmodal associations between visual textures and temperature concepts

Visual textures are critical in how individuals form sensory expectations about objects, which include somatosensory properties such as temperature. This study aimed to uncover crossmodal associations between visual textures and temperature concepts. In Experiment 1 (N = 193), we evaluated crossmodal associations between 43 visual texture categories and different temperature concepts (via temperature words such as cold and hot) using an explicit forced-choice test. The results revealed associations between striped, cracked, matted, and waffled visual textures and high temperatures and between crystalline and flecked visual textures and low temperatures. In Experiment 2 (N = 247), we conducted six implicit association tests (IATs) pairing the two visual textures most strongly associated with low (crystalline and flecked) and high (striped and cracked) temperatures with the words cold and hot as per the results of Experiment 1. When pairing the crystalline and striped visual textures, the results revealed that crystalline was matched to the word cold, and striped was matched to the word hot. However, some associations found in the explicit test were not found in the IATs. In Experiment 3 (N = 124), we investigated how mappings between visual textures and concrete entities may influence crossmodal associations with temperature and these visual textures. Altogether, we found a range of association strengths and automaticity levels. Importantly, we found evidence of relative effects. Furthermore, some of these crossmodal associations are partly influenced by indirect mappings to concrete entities.

Unsupervised learning reveals interpretable latent representations for translucency perception

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is important but challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials, whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference from natural images. Here, we develop an unsupervised style-based image generation model to identify perceptually relevant dimensions for translucent material appearances from photographs. We find our model, with its layer-wise latent representation, can synthesize images of diverse and realistic materials. Importantly, without supervision, human-understandable scene attributes, including the object's shape, material, and body color, spontaneously emerge in the model's layer-wise latent space in a scale-specific manner. By embedding an image into the learned latent space, we can manipulate specific layers' latent code to modify the appearance of the object in the image. Specifically, we find that manipulation on the early-layers (coarse spatial scale) transforms the object's shape, while manipulation on the later-layers (fine spatial scale) modifies its body color. The middle-layers of the latent space selectively encode translucency features and manipulation of such layers coherently modifies the translucency appearance, without changing the object's shape or body color. Moreover, we find the middle-layers of the latent space can successfully predict human translucency ratings, suggesting that translucent impressions are established in mid-to-low spatial scale features. This layer-wise latent representation allows us to systematically discover perceptually relevant image features for human translucency perception. Together, our findings reveal that learning the scale-specific statistical structure of natural images might be crucial for humans to efficiently represent material properties across contexts.

Conditions of a Multi-View 3D Display for Accurate Reproduction of Perceived Glossiness

Visualizing objects as they are perceived in the real world is often critical in our daily experiences. We previously focused on objects' surface glossiness visualized with a 3D display and found that a multi-view 3D display reproduces perceived glossiness more accurately than a 2D display. This improvement of glossiness reproduction can be explained by the fact that a glossy surface visualized by a multi-view 3D display appropriately provides luminance differences between the two eyes and luminance changes accompanying the viewer's lateral head motion. In the present study, to determine the requirements of a multi-view 3D display for the accurate reproduction of perceived glossiness, we developed a simulator of a multi-view 3D display to independently and simultaneously manipulate the viewpoint interval and the magnitude of the optical inter-view crosstalk. Using the simulator, we conducted a psychophysical experiment and found that glossiness reproduction is most accurate when the viewpoint interval is small and there is just a small (but not too small) amount of crosstalk. We proposed a simple yet perceptually valid model that quantitatively predicts the reproduction accuracy of perceived glossiness.

Analysis based on neural representation of natural object surfaces to elucidate the mechanisms of a trained AlexNet model

Analysis and understanding of trained deep neural networks (DNNs) can deepen our understanding of the visual mechanisms involved in primate visual perception. However, due to the limited availability of neural activity data recorded from various cortical areas, the correspondence between the characteristics of artificial and biological neural responses for visually recognizing objects remains unclear at the layer level of DNNs. In the current study, we investigated the relationships between the artificial representations in each layer of a trained AlexNet model (based on a DNN) for object classification and the neural representations in various levels of visual cortices such as the primary visual (V1), intermediate visual (V4), and inferior temporal cortices. Furthermore, we analyzed the profiles of the artificial representations at a single channel level for each layer of the AlexNet model. We found that the artificial representations in the lower-level layers of the trained AlexNet model were strongly correlated with the neural representation in V1, whereas the responses of model neurons in layers at the intermediate and higher-intermediate levels of the trained object classification model exhibited characteristics similar to those of neural activity in V4 neurons. These results suggest that the trained AlexNet model may gradually establish artificial representations for object classification through the hierarchy of its network, in a similar manner to the neural mechanisms by which afferent transmission beginning in the low-level features gradually establishes object recognition as signals progress through the hierarchy of the ventral visual pathway.

Identifying specular highlights: Insights from deep learning

Specular highlights are the most important image feature for surface gloss perception. Yet, recognizing whether a bright patch in an image is due to specular reflection or some other cause (e.g., texture marking) is challenging, and it remains unclear how the visual system reliably identifies highlights. There is currently no image-computable model that emulates human highlight identification, so here we sought to develop a neural network that reproduces observers' characteristic successes and failures. We rendered 179,085 images of glossy, undulating, textured surfaces. Given such images as input, a feedforward convolutional neural network was trained to output an image containing only the specular reflectance component. Participants viewed such images and reported whether or not specific pixels were highlights. The queried pixels were carefully selected to distinguish between ground truth and a simple thresholding of image intensity. The neural network outperformed the simple thresholding model-and ground truth-at predicting human responses. We then used a genetic algorithm to selectively delete connections within the neural network to identify variants of the network that approximated human judgments even more closely. The best resulting network shared 68% of the variance with human judgments-more than the unpruned network. As a first step toward interpreting the network, we then used representational similarity analysis to compare its inner representations to a wide variety of hand-engineered image features. We find that the network learns representations that are similar not only to directly image-computable predictors but also to more complex predictors such as intrinsic or geometric factors, as well as some indications of photo-geometrical constraints learned by the network. However, our network fails to replicate human response patterns to violations of photo-geometric constraints (rotated highlights) as described by other authors.

Tactile information affects alternating visual percepts during binocular rivalry using naturalistic objects

INTRODUCTION

Past studies have provided evidence that the effects of tactile stimulation on binocular rivalry are mediated by primitive features (orientation and spatial frequency) common in vision and touch. In this study, we examined whether such effects on binocular rivalry can be obtained through the roughness of naturalistic objects. In three experiments, the total dominant time of visual percepts of two objects was measured under binocular rivalry when participants touched one of the objects.

RESULT

In Experiment 1, the total dominant time for the image of artificial turf and bathmat was prolonged by congruent tactile stimulation and shortened by incongruent tactile stimulation. In Experiment 2, we used the same stimuli but rotated their visual images in opposite directions. The dominant time for either image was prolonged by congruent tactile stimulation. In Experiment 3, we used different types of stimuli, smooth marble and rough fabric, and noted significant effects of the congruent and incongruent tactile stimulation on the dominant time of visual percepts.

CONCLUSION

These three experiments demonstrated that visuo-tactile interaction on binocular rivalry can be mediated by roughness.

Distinguishing mirror from glass: A "big data" approach to material perception

Distinguishing mirror from glass is a challenging visual inference, because both materials derive their appearance from their surroundings, yet we rarely experience difficulties in telling them apart. Very few studies have investigated how the visual system distinguishes reflections from refractions and to date, there is no image-computable model that emulates human judgments. Here we sought to develop a deep neural network that reproduces the patterns of visual judgments human observers make. To do this, we trained thousands of convolutional neural networks on more than 750,000 simulated mirror and glass objects, and compared their performance with human judgments, as well as alternative classifiers based on "hand-engineered" image features. For randomly chosen images, all classifiers and humans performed with high accuracy, and therefore correlated highly with one another. However, to assess how similar models are to humans, it is not sufficient to compare accuracy or correlation on random images. A good model should also predict the characteristic errors that humans make. We, therefore, painstakingly assembled a diagnostic image set for which humans make systematic errors, allowing us to isolate signatures of human-like performance. A large-scale, systematic search through feedforward neural architectures revealed that relatively shallow (three-layer) networks predicted human judgments better than any other models we tested. This is the first image-computable model that emulates human errors and succeeds in distinguishing mirror from glass, and hints that mid-level visual processing might be particularly important for the task.

Gloss perception: Searching for a deep neural network that behaves like humans

The visual computations underlying human gloss perception remain poorly understood, and to date there is no image-computable model that reproduces human gloss judgments independent of shape and viewing conditions. Such a model could provide a powerful platform for testing hypotheses about the detailed workings of surface perception. Here, we made use of recent developments in artificial neural networks to test how well we could recreate human responses in a high-gloss versus low-gloss discrimination task. We rendered >70,000 scenes depicting familiar objects made of either mirror-like or near-matte textured materials. We trained numerous classifiers to distinguish the two materials in our images-ranging from linear classifiers using simple pixel statistics to convolutional neural networks (CNNs) with up to 12 layers-and compared their classifications with human judgments. To determine which classifiers made the same kinds of errors as humans, we painstakingly identified a set of 60 images in which human judgments are consistently decoupled from ground truth. We then conducted a Bayesian hyperparameter search to identify which out of several thousand CNNs most resembled humans. We found that, although architecture has only a relatively weak effect, high correlations with humans are somewhat more typical in networks of shallower to intermediate depths (three to five layers). We also trained deep convolutional generative adversarial networks (DCGANs) of different depths to recreate images based on our high- and low-gloss database. Responses from human observers show that two layers in a DCGAN can recreate gloss recognizably for human observers. Together, our results indicate that human gloss classification can best be explained by computations resembling early to mid-level vision.

Computer Vision System for Expressing Texture Using Sound-Symbolic Words

The major goals of texture research in computer vision are to understand, model, and process texture and ultimately simulate human visual information processing using computer technologies. The field of computer vision has witnessed remarkable advancements in material recognition using deep convolutional neural networks (DCNNs), which have enabled various computer vision applications, such as self-driving cars, facial and gesture recognition, and automatic number plate recognition. However, for computer vision to "express" texture like human beings is still difficult because texture description has no correct or incorrect answer and is ambiguous. In this paper, we develop a computer vision method using DCNN that expresses texture of materials. To achieve this goal, we focus on Japanese "sound-symbolic" words, which can describe differences in texture sensation at a fine resolution and are known to have strong and systematic sensory-sound associations. Because the phonemes of Japanese sound-symbolic words characterize categories of texture sensations, we develop a computer vision method to generate the phonemes and structure comprising sound-symbolic words that probabilistically correspond to the input images. It was confirmed that the sound-symbolic words output by our system had about 80% accuracy rate in our evaluation.

The visual prefrontal cortex of anthropoids: interaction with temporal cortex in decision making and its role in the making of "visual animals"

The ventral prefrontal cortex (PFC) of primates-a region strongly implicated in decision making-receives highly processed visual sensory inputs from the inferior temporal cortex (ITC) and perirhinal cortex (PRC) and can therefore be considered visual PFC. Usually, the functions of temporal cortex and visual PFC have been discussed in separate literatures. By considering them together, we aim to clarify the ways in which fronto-temporal networks guide decision making. After discussing the ways in which visual PFC interacts with temporal cortex to promote decision making, we offer specific predictions about the selective roles of the ITC- and PRC-based fronto-temporal networks. Finally, we suggest that an increased reliance on visual PFC in anthropoid primates led to our emergence as 'visual' animals.

Metallic: A Bivalent Ambimodal Material Property?

Many metallic visual stimuli, especially the so-called precious metals, have long had a rich symbolic meaning for humans. Intriguingly, however, while metallic is used to describe sensations associated with pretty much every sensory modality, the descriptor is normally positively valenced in the case of vision while typically being negatively valenced in the case of those metallic sensations that are elicited by the stimulation of the chemical senses. In fact, outside the visual modality, metallic would often appear to be used to describe those sensations that are unfamiliar and unpleasant as much as to refer to any identifiable perceptual quality (or attribute). In this review, we assess those sensory stimuli that people choose to refer to as metallic, summarising the multiple, often symbolic, meanings of (especially precious) metals. The evidence of positively valenced sensation transference from metallic serviceware (e.g., plates, cups, and cutlery) to the food and drink with which it comes into contact is also reviewed.

Structure and function of neural circuit related to gloss perception in the macaque inferior temporal cortex: a case report

The inferior temporal (IT) cortex of the macaque monkey plays a pivotal role in the visual recognition of objects. In the IT cortex, a feature-selective network formed by connecting subregions specialized for common visual features seems to be a basic strategy for processing biologically important visual features. Gloss perception plays an important role in the judgment of materials and conditions of objects and is a biologically significant visual function. In the present study, we attempted to determine whether a neural circuit specialized for processing information related to gloss perception exists in the IT cortex in one monkey. We injected retrograde tracer into a gloss-selective subregion in the IT cortex where gloss-selective neurons were clustered in the neural recording experiment, and anatomically examined its neural connections. We observed that retrogradely labeled neurons were densely accumulated in multiple locations in the posterior and anterior IT cortices. Based on the results of this case study, we will discuss the possibility that, together with the injection site, the sites with a dense cluster of labeled neurons form feature-selective neural circuits for the processing of gloss information in the IT cortex.

On the Questionable Appeal of Glossy/Shiny Food Packaging

Those stimuli that have a shiny/glossy visual appearance are typically rated as both attractive and attention capturing. Indeed, for millennia, shiny precious metals and glossy lacquerware have been used to enhance the presentation, and thus the perception, of food and drink. As such, one might have expected that adding a shiny/glossy appearance/finish to the outer packaging of food and beverage products would also be desirable. However, the latest research appears to show that many consumers have internalised an association between glossy packaging and greasy (or unhealthy) food products, while matte packaging tends to be associated with those foods that are more natural instead. Furthermore, it turns out that many consumers do not necessarily appreciate the attempt to capture their attention that glossy packaging so often affords. At the same time, it is important to recognise that somewhat different associations may apply in the case of inner versus outer food and beverage packaging. Shiny metallic (inner) packaging may well prime (rightly or wrongly) concerns about sustainability amongst consumers. Given the research that has been published in recent years, food and beverage manufacturers/marketers should think very carefully about whether or not to introduce such shiny/glossy finishes to their packaging.

Dynamic dot displays reveal material motion network in the human brain

There is growing research interest in the neural mechanisms underlying the recognition of material categories and properties. This research field, however, is relatively more recent and limited compared to investigations of the neural mechanisms underlying object and scene category recognition. Motion is particularly important for the perception of non-rigid materials, but the neural basis of non-rigid material motion remains unexplored. Using fMRI, we investigated which brain regions respond preferentially to material motion versus other types of motion. We introduce a new database of stimuli - dynamic dot materials - that are animations of moving dots that induce vivid percepts of various materials in motion, e.g. flapping cloth, liquid waves, wobbling jelly. Control stimuli were scrambled versions of these same animations and rigid three-dimensional rotating dots. Results showed that isolating material motion properties with dynamic dots (in contrast with other kinds of motion) activates a network of cortical regions in both ventral and dorsal visual pathways, including areas normally associated with the processing of surface properties and shape, and extending to somatosensory and premotor cortices. We suggest that such a widespread preference for material motion is due to strong associations between stimulus properties. For example viewing dots moving in a specific pattern not only elicits percepts of material motion; one perceives a flexible, non-rigid shape, identifies the object as a cloth flapping in the wind, infers the object's weight under gravity, and anticipates how it would feel to reach out and touch the material. These results are a first important step in mapping out the cortical architecture and dynamics in material-related motion processing.

Expectations affect the perception of material properties

Many objects that we encounter have typical material qualities: spoons are hard, pillows are soft, and Jell-O dessert is wobbly. Over a lifetime of experiences, strong associations between an object and its typical material properties may be formed, and these associations not only include how glossy, rough, or pink an object is, but also how it behaves under force: we expect knocked over vases to shatter, popped bike tires to deflate, and gooey grilled cheese to hang between two slices of bread when pulled apart. Here we ask how such rich visual priors affect the visual perception of material qualities and present a particularly striking example of expectation violation. In a cue conflict design, we pair computer-rendered familiar objects with surprising material behaviors (a linen curtain shattering, a porcelain teacup wrinkling, etc.) and find that material qualities are not solely estimated from the object's kinematics (i.e., its physical [atypical] motion while shattering, wrinkling, wobbling etc.); rather, material appearance is sometimes “pulled” toward the “native” motion, shape, and optical properties that are associated with this object. Our results, in addition to patterns we find in response time data, suggest that visual priors about materials can set up high-level expectations about complex future states of an object and show how these priors modulate material appearance.

Visual Design Cues Impacting Food Choice: A Review and Future Research Agenda

This review aims to tackle the challenge of understanding how visual design cues can affect behavioural outcomes in a food context. The review answers two key questions: (1) What are the effects of the most important visual design cues on behavioural outcomes and how can they be explained? (2) What are the research gaps in this area? We start from a comprehensive taxonomy of visual design cues delineating the most important visual design cues. Next, we evaluate the extant research based on a structured, narrative literature review on visual design cues in the food domain. We differentiate between object processed and spatially processed visual design cues in food choice contexts and show how they affect behavioural outcomes through a range of psychological processes (attention, affective-, cognitive- and motivational reactions, food perceptions and attitudes). We end with recommendations which take into account the current food store context, the state-of-art in measuring psychological processes and behavioural outcomes and the specific food-, person- and context-related moderators. This review offers guidance for research to untangle the complexity of the effect of visual design cues in a food choice context.

Material constancy in perception and working memory

A key challenge for the visual system entails the extraction of constant properties of objects from sensory information that varies moment by moment due to changes in viewing conditions. Although successful performance in constancy tasks requires cooperation between perception and working memory, the function of the memory system has been under-represented in recent material perception literature. Here, we addressed the limits of material constancy by elucidating if and how working memory is involved in constancy tasks by using a variety of material stimuli, such as metals, glass, and translucent objects. We conducted experiments with a simultaneous and a successive matching-to-sample paradigm in which participants matched the perceived material properties of objects with or without a temporal delay under varying illumination contexts. The current study combined a detailed analysis of matching errors, data on the strategy use obtained via a self-report questionnaire, and the statistical image analysis of diagnostic image cues used for material discrimination. We found a comparable material constancy between simultaneous and successive matching conditions, and it was suggested that, in both matching conditions, participants used similar information processing strategies for the discrimination of materials. The study provides converging evidence on the critical role of working memory in material constancy, where working memory serves as a shared processing bottleneck that constrains both simultaneous and successive material constancy.

The limited contribution of early visual cortex in visual working memory for surface roughness

Are visual representations in the human early visual cortex necessary for visual working memory (VWM)? Previous studies suggest that VWM is underpinned by distributed representations across several brain regions, including the early visual cortex. Notably, in these studies, participants had to memorize images under consistent visual conditions. However, in our daily lives, we must retain the essential visual properties of objects despite changes in illumination or viewpoint. The role of brain regions-particularly the early visual cortices-in these situations remains unclear. The present study investigated whether the early visual cortex was essential for achieving stable VWM. Focusing on VWM for object surface properties, we conducted fMRI experiments, while male and female participants performed a delayed roughness discrimination task in which sample and probe spheres were presented under varying illumination. By applying multi-voxel pattern analysis to brain activity in regions of interest, we found that the ventral visual cortex and intraparietal sulcus were involved in roughness VWM under changing illumination conditions. In contrast, VWM was not supported as robustly by the early visual cortex. These findings show that visual representations in the early visual cortex alone are insufficient for the robust roughness VWM representation required during changes in illumination.

Senses of place: architectural design for the multisensory mind

Traditionally, architectural practice has been dominated by the eye/sight. In recent decades, though, architects and designers have increasingly started to consider the other senses, namely sound, touch (including proprioception, kinesthesis, and the vestibular sense), smell, and on rare occasions, even taste in their work. As yet, there has been little recognition of the growing understanding of the multisensory nature of the human mind that has emerged from the field of cognitive neuroscience research. This review therefore provides a summary of the role of the human senses in architectural design practice, both when considered individually and, more importantly, when studied collectively. For it is only by recognizing the fundamentally multisensory nature of perception that one can really hope to explain a number of surprising crossmodal environmental or atmospheric interactions, such as between lighting colour and thermal comfort and between sound and the perceived safety of public space. At the same time, however, the contemporary focus on synaesthetic design needs to be reframed in terms of the crossmodal correspondences and multisensory integration, at least if the most is to be made of multisensory interactions and synergies that have been uncovered in recent years. Looking to the future, the hope is that architectural design practice will increasingly incorporate our growing understanding of the human senses, and how they influence one another. Such a multisensory approach will hopefully lead to the development of buildings and urban spaces that do a better job of promoting our social, cognitive, and emotional development, rather than hindering it, as has too often been the case previously.

Shitsukan - the Multisensory Perception of Quality

We often estimate, or perceive, the quality of materials, surfaces, and objects, what the Japanese refer to as 'shitsukan', by means of several of our senses. The majority of the literature on shitsukan perception has, though, tended to focus on the unimodal visual evaluation of stimulus properties. In part, this presumably reflects the widespread hegemony of the visual in the modern era and, in part, is a result of the growing interest, not to mention the impressive advances, in digital rendering amongst the computer graphics community. Nevertheless, regardless of such an oculocentric bias in so much of the empirical literature, it is important to note that several other senses often do contribute to the impression of the material quality of surfaces, materials, and objects as experienced in the real world, rather than just in virtual reality. Understanding the multisensory contributions to the perception of material quality, especially when combined with computational and neural data, is likely to have implications for a number of fields of basic research as well as being applicable to emerging domains such as, for example, multisensory augmented retail, not to mention multisensory packaging design.

Fats are Glossy but Does Glossiness Imply Fatness? The Influence of Packaging Glossiness on Food Perceptions

This research brings together two research streams, one focusing on the influence of a diverse set of packaging attributes (e.g., shape, size, color, etc.) on perceptions of packaged food and the second one on the up- and downsides of using glossy materials, which are often studied in a non-food context. The current research deals with the influence of glossy (versus matte) food packages on consumers’ perceptions of the food inside the package. With one online survey and one quasi-experiment, we show that consumers draw inferences on the food’s fat level from the package surface, in that glossy packages are seen as a signal of fatness. This association is specific; consumers do not associate glossiness with every unhealthy product aspect. Sugar levels are unaffected by the package surface. However, due to the higher inferred fat level, a product in a glossy package is perceived to be less healthy, less tasty, and low in quality and product expensiveness. Thus, these findings suggest that glossy (versus matte) food packages mainly serve as a signal of negative product qualities.

Neural correlates of visual short-term memory for objects with material categories

Behavioral and neuroscience studies have shown that we can easily identify material categories, such as metal and fabric. Not only the early visual areas but also higher-order visual areas including the fusiform gyrus are known to be engaged in material perception. However, the brain mechanisms underlying visual short-term memory (VSTM) for material categories are unknown. To address this issue, we examined the neural correlates of VSTM for objects with material categories using a change detection task. In each trial, participants viewed a sample display containing two, four, or six objects having six material categories and were required to remember the locations and types of objects. After a brief delay, participants were asked to detect an object change based on the images or material categories in the test display (image-based and material-based conditions). Neuronal activity in the brain was assessed using functional magnetic resonance imaging (MRI). Behavioral results showed that the number of objects encoded did not increase as a function of set size in either image-based or material-based conditions. By contrast, MRI data showed a difference between the image-based and material-based conditions in percent signal change observed in a priori region of interest, the fusiform face area (FFA). Thus, we failed to achieve our research aim. However, the brain activation in the FFA correlated with the activation in the precentral/postcentral gyrus, which is related to haptic processing. Our findings indicate that the FFA may be involved in VSTM for objects with material categories in terms of the difference between images and material categories and that this memory may be mediated by the tactile properties of objects.