The look and feel of soft are similar across different softness dimensions

Probing the link between vision and language in material perception using psychophysics and unsupervised learning

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to describe what we see and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression to understand how visual features relate to semantic representations in human cognition. We use deep generative models to generate images of realistic materials. Interpolating between the generative models enables us to systematically create material appearances in both well-defined and ambiguous categories. Using these stimuli, we compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among ambiguous materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual nuances of material appearances. Analyzing the image representation of materials obtained from various pre-trained deep neural networks, we find that similarity structures in human visual judgments align more closely with those of the vision-language models than purely vision-based models. Our work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

Visual softness perception can be manipulated through exploratory procedures

Both visual and haptic softness perception have recently been shown to have multiple dimensions, such as deformability, granularity, fluidity, surface softness, and roughness. During haptic exploration, people adjust their hand motions (exploratory procedures, EPs) based on the material qualities of the object and the particular information they intend to acquire. Some of these EPs are also shown to be associated with perceived softness dimensions, for example, stroking a silk blouse or applying pressure to a pillow. Here, we aimed to investigate whether we can manipulate observers' judgments about softness attributes through exposure to videos of others performing various EPs on everyday soft materials. In two experiments, participants watched two videos of the same material: one with a corresponding EP and the other without correspondence; then, they judged these materials based on 12 softness-related adjectives (semantic differentiation method). The results of the second experiment suggested that when the EP is congruent with the dimension from which the material is chosen, the ratings for the adjectives from the same dimension are higher than the incongruent EP. This study provides evidence that participants can assess material properties from optic and mechanical cues without needing haptic signals. Additionally, our findings indicate that manipulating the hand motion can selectively facilitate material-related judgments.

Probing the Link Between Vision and Language in Material Perception Using Psychophysics and Unsupervised Learning

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to express our subjective understanding of visual input and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression in material perception to understand how visual features relate to semantic representations. We use deep generative networks to construct an expandable image space to systematically create materials of well-defined and ambiguous categories. From such a space, we sampled diverse stimuli and compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual features that characterize the material's optical properties. Analyzing the image representation of materials obtained from various pre-trained data-rich deep neural networks, we find that human visual judgments' similarity structures align more closely with those of the text-guided visual-semantic model than purely vision-based models. Our findings suggest that while semantic representations facilitate material categorization, non-semantic visual features also play a significant role in discriminating materials at a finer level. This work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for quantitatively evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

SORI: A softness-rendering interface to unravel the nature of softness perception

Tactile perception of softness serves a critical role in the survival, well-being, and social interaction among various species, including humans. This perception informs activities from food selection in animals to medical palpation for disease detection in humans. Despite its fundamental importance, a comprehensive understanding of how softness is neurologically and cognitively processed remains elusive. Previous research has demonstrated that the somatosensory system leverages both cutaneous and kinesthetic cues for the sensation of softness. Factors such as contact area, depth, and force play a particularly critical role in sensations experienced at the fingertips. Yet, existing haptic technologies designed to explore this phenomenon are limited, as they often couple force and contact area, failing to provide a real-world experience of softness perception. Our research introduces the softness-rendering interface (SORI), a haptic softness display designed to bridge this knowledge gap. Unlike its predecessors, SORI has the unique ability to decouple contact area and force, thereby allowing for a quantitative representation of softness sensations at the fingertips. Furthermore, SORI incorporates individual physical fingertip properties and model-based softness cue estimation and mapping to provide a highly personalized experience. Utilizing this method, SORI quantitatively replicates the sensation of softness on stationary, dynamic, homogeneous, and heterogeneous surfaces. We demonstrate that SORI accurately renders the surfaces of both virtual and daily objects, thereby presenting opportunities across a range of fields, from teleoperation to medical technology. Finally, our proposed method and SORI will expedite psychological and neuroscience research to unlock the nature of softness perception.

Selectively manipulating softness perception of materials through sound symbolism

Cross-modal interactions between auditory and haptic perception manifest themselves in language, such as sound symbolic words: crunch, splash, and creak. Several studies have shown strong associations between sound symbolic words, shapes (e.g., Bouba/Kiki effect), and materials. Here, we identified these material associations in Turkish sound symbolic words and then tested for their effect on softness perception. First, we used a rating task in a semantic differentiation method to extract the perceived softness dimensions from words and materials. We then tested whether Turkish onomatopoeic words can be used to manipulate the perceived softness of everyday materials such as honey, silk, or sand across different dimensions of softness. In the first preliminary study, we used 40 material videos and 29 adjectives in a rating task with a semantic differentiation method to extract the main softness dimensions. A principal component analysis revealed seven softness components, including Deformability, Viscosity, Surface Softness, and Granularity, in line with the literature. The second preliminary study used 27 onomatopoeic words and 21 adjectives in the same rating task. Again, the findings aligned with the literature, revealing dimensions such as Viscosity, Granularity, and Surface Softness. However, no factors related to Deformability were found due to the absence of sound symbolic words in this category. Next, we paired the onomatopoeic words and material videos based on their associations with each softness dimension. We conducted a new rating task, synchronously presenting material videos and spoken onomatopoeic words. We hypothesized that congruent word-video pairs would produce significantly higher ratings for dimension-related adjectives, while incongruent word-video pairs would decrease these ratings, and the ratings of unrelated adjectives would remain the same. Our results revealed that onomatopoeic words selectively alter the perceived material qualities, providing evidence and insight into the cross-modality of perceived softness.

Assessing the representational structure of softness activated by words

Softness is an important material property that can be judged directly, by interacting with an object, but also indirectly, by simply looking at an image of a material. The latter is likely possible by filling in relevant multisensory information from prior experiences with soft materials. Such experiences are thought to lead to associations that make up our representations about perceptual softness. Here, we investigate the structure of this representational space when activated by words, and compare it to haptic and visual perceptual spaces that we obtained in earlier work. To this end, we performed an online study where people rated different sensory aspects of soft materials, presented as written names. We compared the results with the previous studies where identical ratings were made on the basis of visual and haptic information. Correlation and Procrustes analyses show that, overall, the representational spaces of verbally presented materials were similar to those obtained from haptic and visual experiments. However, a classifier analysis showed that verbal representations could better be predicted from those obtained from visual than from haptic experiments. In a second study we rule out that these larger discrepancies in representations between verbal and haptic conditions could be due to difficulties in material identification in haptic experiments. We discuss the results with respect to the recent idea that at perceived softness is a multidimensional construct.

Roughness perception: A multisensory/crossmodal perspective

Roughness is a perceptual attribute typically associated with certain stimuli that are presented in one of the spatial senses. In auditory research, the term is typically used to describe the harsh effects that are induced by particular sound qualities (i.e., dissonance) and human/animal vocalizations (e.g., screams, distress cries). In the tactile domain, roughness is a crucial factor determining the perceptual features of a surface. The same feature can also be ascertained visually, by means of the extraction of pattern features that determine the haptic quality of surfaces, such as grain size and density. By contrast, the term roughness has rarely been applied to the description of those stimuli perceived via the chemical senses. In this review, we take a critical look at the putative meaning(s) of the term roughness, when used in both unisensory and multisensory contexts, in an attempt to answer two key questions: (1) Is the use of the term 'roughness' the same in each modality when considered individually? and (2) Do crossmodal correspondences involving roughness match distinct perceptual features or (at least on certain occasions) do they merely pick-up on an amodal property? We start by examining the use of the term in the auditory domain. Next, we summarize the ways in which the term roughness has been used in the literature on tactile and visual perception, and in the domain of olfaction and gustation. Then, we move on to the crossmodal context, reviewing the literature on the perception of roughness in the audiovisual, audiotactile, and auditory-gustatory/olfactory domains. Finally, we highlight some limitations of the reviewed literature and we outline a number of key directions for future empirical research in roughness perception.