Learning from vision-to-touch is different than learning from touch-to-vision

Meaning of Gaze Behaviors in Individuals' Perception and Interpretation of Commercial Interior Environments: An Experimental Phenomenology Approach Involving Eye-Tracking

A critical question in interior design is how multisensory information is integrated into occupant perception and interpretation of the environmental contexts and meanings. Although there have been efforts to identify and theorize visual perception of interior factors or features (e.g., colors, fixtures, and signs), the hidden meanings behind visual attention and behaviors have been neglected in interior design research. This experimental phenomenological study investigates the impact of auditory stimuli on the gaze behaviors of individuals and the hidden meanings of their audio-visual perceptions of commercial interiors. Implementing eye-tracking and open-ended interviews, this study explored how the neurophysiological and phenomenological methods in complementary can serve for interior design research on the meaning of gaze behaviors. The study used a convenience sample of 26 participants, three coffee shop interior images, and two musical stimuli. Essential to this study is the interpretive analysis of corresponding eye-tracking and interview data. The results show that visual perception is affected by auditory stimuli and other interior elements and factors associated with personal experiences; however, no distinct gaze pattern is identified by the type of auditory stimuli. The fixation patterns showed mixed reflections of the participants' perceptions, e.g., a single fixation pattern reflecting participants' likes and dislikes. Findings included six essential meanings of participants' gaze behaviors. This study suggested that auditory and visual stimuli are reciprocal in individuals' perceptions. Rather than one affects the other, the interaction between sensory stimuli contributes to the complexity and intensity of multisensory stimuli people associate with their experiences and conceptualize with meanings they establish.

Soft like velvet and shiny like satin: Perceptual material signatures of fabrics depicted in 17th century paintings

Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights' coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.

Modality-specific attention attenuates visual-tactile integration and recalibration effects by reducing prior expectations of a common source for vision and touch

At any moment in time, streams of information reach the brain through the different senses. Given this wealth of noisy information, it is essential that we select information of relevance - a function fulfilled by attention - and infer its causal structure to eventually take advantage of redundancies across the senses. Yet, the role of selective attention during causal inference in cross-modal perception is unknown. We tested experimentally whether the distribution of attention across vision and touch enhances cross-modal spatial integration (visual-tactile ventriloquism effect, Expt. 1) and recalibration (visual-tactile ventriloquism aftereffect, Expt. 2) compared to modality-specific attention, and then used causal-inference modeling to isolate the mechanisms behind the attentional modulation. In both experiments, we found stronger effects of vision on touch under distributed than under modality-specific attention. Model comparison confirmed that participants used Bayes-optimal causal inference to localize visual and tactile stimuli presented as part of a visual-tactile stimulus pair, whereas simultaneously collected unity judgments - indicating whether the visual-tactile pair was perceived as spatially-aligned - relied on a sub-optimal heuristic. The best-fitting model revealed that attention modulated sensory and cognitive components of causal inference. First, distributed attention led to an increase of sensory noise compared to selective attention toward one modality. Second, attending to both modalities strengthened the stimulus-independent expectation that the two signals belong together, the prior probability of a common source for vision and touch. Yet, only the increase in the expectation of vision and touch sharing a common source was able to explain the observed enhancement of visual-tactile integration and recalibration effects with distributed attention. In contrast, the change in sensory noise explained only a fraction of the observed enhancements, as its consequences vary with the overall level of noise and stimulus congruency. Increased sensory noise leads to enhanced integration effects for visual-tactile pairs with a large spatial discrepancy, but reduced integration effects for stimuli with a small or no cross-modal discrepancy. In sum, our study indicates a weak a priori association between visual and tactile spatial signals that can be strengthened by distributing attention across both modalities.

Optimal visual-haptic integration with articulated tools

When we feel and see an object, the nervous system integrates visual and haptic information optimally, exploiting the redundancy in multiple signals to estimate properties more precisely than is possible from either signal alone. We examined whether optimal integration is similarly achieved when using articulated tools. Such tools (tongs, pliers, etc) are a defining characteristic of human hand function, but complicate the classical sensory 'correspondence problem' underlying multisensory integration. Optimal integration requires establishing the relationship between signals acquired by different sensors (hand and eye) and, therefore, in fundamentally unrelated units. The system must also determine when signals refer to the same property of the world-seeing and feeling the same thing-and only integrate those that do. This could be achieved by comparing the pattern of current visual and haptic input to known statistics of their normal relationship. Articulated tools disrupt this relationship, however, by altering the geometrical relationship between object properties and hand posture (the haptic signal). We examined whether different tool configurations are taken into account in visual-haptic integration. We indexed integration by measuring the precision of size estimates, and compared our results to optimal predictions from a maximum-likelihood integrator. Integration was near optimal, independent of tool configuration/hand posture, provided that visual and haptic signals referred to the same object in the world. Thus, sensory correspondence was determined correctly (trial-by-trial), taking tool configuration into account. This reveals highly flexible multisensory integration underlying tool use, consistent with the brain constructing internal models of tools' properties.

Mental Reactivation and Pleasantness Judgment of Experience Related to Vision, Hearing, Skin Sensations, Taste and Olfaction

Language acquisition is based on our knowledge about the world and forms through multiple sensory-motor interactions with the environment. We link the properties of individual experience formed at different stages of ontogeny with the phased development of sensory modalities and with the acquisition of words describing the appropriate forms of sensitivity. To test whether early-formed experience related to skin sensations, olfaction and taste differs from later-formed experience related to vision and hearing, we asked Russian-speaking participants to categorize or to assess the pleasantness of experience mentally reactivated by sense-related adjectives found in common dictionaries. It was found that categorizing adjectives in relation to vision, hearing and skin sensations took longer than categorizing adjectives in relation to olfaction and taste. In addition, experience described by adjectives predominantly related to vision, hearing and skin sensations took more time for the pleasantness judgment and generated less intense emotions than that described by adjectives predominantly related to olfaction and taste. Interestingly the dynamics of skin resistance corresponded to the intensity and pleasantness of reported emotions. We also found that sense-related experience described by early-acquired adjectives took less time for the pleasantness judgment and generated more intense and more positive emotions than that described by later-acquired adjectives. Correlations were found between the time of the pleasantness judgment of experience, intensity and pleasantness of reported emotions, age of acquisition, frequency, imageability and length of sense-related adjectives. All in all these findings support the hypothesis that early-formed experience is less differentiated than later-formed experience.

Touch influences perceived gloss

Identifying an object's material properties supports recognition and action planning: we grasp objects according to how heavy, hard or slippery we expect them to be. Visual cues to material qualities such as gloss have recently received attention, but how they interact with haptic (touch) information has been largely overlooked. Here, we show that touch modulates gloss perception: objects that feel slippery are perceived as glossier (more shiny).Participants explored virtual objects that varied in look and feel. A discrimination paradigm (Experiment 1) revealed that observers integrate visual gloss with haptic information. Observers could easily detect an increase in glossiness when it was paired with a decrease in friction. In contrast, increased glossiness coupled with decreased slipperiness produced a small perceptual change: the visual and haptic changes counteracted each other. Subjective ratings (Experiment 2) reflected a similar interaction - slippery objects were rated as glossier and vice versa. The sensory system treats visual gloss and haptic friction as correlated cues to surface material. Although friction is not a perfect predictor of gloss, the visual system appears to know and use a probabilistic relationship between these variables to bias perception - a sensible strategy given the ambiguity of visual clues to gloss.