Are visual texture-selective areas recruited during haptic texture discrimination?

How to construct liquid-crystal spectacles to control vision of real-world objects and environments

A major challenge in studying naturalistic vision lies in controlling stimulus and scene viewing time. This is especially the case for studies using real-world objects as stimuli (rather than computerized images) because real objects cannot be "onset" and "offset" in the same way that images can be. Since the late 1980s, one solution to this problem has been to have the observer wear electro-optic spectacles with computer-controlled liquid-crystal lenses that switch between transparent ("open") and translucent ("closed") states. Unfortunately, the commercially available glasses (PLATO Visual Occlusion Spectacles) command a high price tag, the hardware is fragile, and the glasses cannot be customized. This led us to explore how to manufacture liquid-crystal occlusion glasses in our own laboratory. Here, we share the products of our work by providing step-by-step instructions for researchers to design, build, operate, and test liquid-crystal glasses for use in experimental contexts. The glasses can be assembled with minimal technical knowledge using readily available components, and they can be customized for different populations and applications. The glasses are robust, and they can be produced at a fraction of the cost of commercial alternatives. Tests of reliability and temporal accuracy show that the performance of our laboratory prototype was comparable to that of the PLATO glasses. We discuss the results of our work with respect to implications for promoting rigor and reproducibility, potential use cases, comparisons with other liquid-crystal shutter glasses, and how users can find information regarding future updates and developments.

The neural correlates of texture perception: A systematic review and activation likelihood estimation meta-analysis of functional magnetic resonance imaging studies

INTRODUCTION

Humans use discriminative touch to perceive texture through dynamic interactions with surfaces, activating low-threshold mechanoreceptors in the skin. It was largely assumed that texture was processed in primary somatosensory regions in the brain; however, imaging studies indicate heterogeneous patterns of brain activity associated with texture processing.

METHODS

To address this, we conducted a coordinate-based activation likelihood estimation meta-analysis of 13 functional magnetic resonance imaging studies (comprising 15 experiments contributing 228 participants and 275 foci) selected by a systematic review.

RESULTS

Concordant activations for texture perception occurred in the left primary somatosensory and motor regions, with bilateral activations in the secondary somatosensory, posterior insula, and premotor and supplementary motor cortices. We also evaluated differences between studies that compared touch processing to non-haptic control (e.g., rest or visual control) or those that used haptic control (e.g., shape or orientation perception) to specifically investigate texture encoding. Studies employing a haptic control revealed concordance for texture processing only in the left secondary somatosensory cortex. Contrast analyses demonstrated greater concordance of activations in the left primary somatosensory regions and inferior parietal cortex for studies with a non-haptic control, compared to experiments accounting for other haptic aspects.

CONCLUSION

These findings suggest that texture processing may recruit higher order integrative structures, and the secondary somatosensory cortex may play a key role in encoding textural properties. The present study provides unique insight into the neural correlates of texture-related processing by assessing the influence of non-textural haptic elements and identifies opportunities for a future research design to understand the neural processing of texture.

Pneumatic tactile stimulus delivery system for studying brain responses evoked by active finger touch with fMRI

BACKGROUND

Primates use their hands to actively touch objects and collect information. To study tactile information processing, it is important for participants to experience tactile stimuli through active touch while monitoring brain activities.

NEW METHOD

Here, we developed a pneumatic tactile stimulus delivery system (pTDS) that delivers various tactile stimuli on a programmed schedule and allows voluntary finger touches during MRI scanning. The pTDS uses a pneumatic actuator to move tactile stimuli and place them in a finger hole. A photosensor detects the time when an index finger touches a tactile stimulus, enabling the analysis of the touch-elicited brain responses.

RESULTS

We examined brain responses while the participants actively touched braille objects presented by the pTDS. BOLD responses during tactile perception were significantly stronger in a finger touch area of the contralateral somatosensory cortex compared with that of visual perception.

CONCLUSION

The pTDS enables MR studies of brain mechanisms for tactile processes through natural finger touch.

Establishing haptic texture attribute space and predicting haptic attributes from image features using 1D-CNN

The current study strives to provide a haptic attribute space where texture surfaces are located based on their haptic attributes. The main aim of the haptic attribute space is to come up with a standardized model for representing and identifying haptic textures analogous to the RGB model for colors. To this end, a four dimensional haptic attribute space is established by conducting a psychophysical experiment where human participants rate 100 real-life texture surfaces according to their haptic attributes. The four dimensions of the haptic attribute space are rough-smooth, flat-bumpy, sticky-slippery, and hard-soft. The generalization and scalability of the haptic attribute space is achieved by training a 1D-CNN model for predicting attributes of haptic textures. The 1D-CNN is trained using the attribute data from psychophysical experiments and image features collected from the images of real textures. The prediction power granted by the 1D-CNN renders scalability to the haptic attribute space. The prediction accuracy of the proposed 1D-CNN model is compared against other machine learning and deep learning algorithms. The results show that the proposed method outperforms the other models on MAE and RMSE metrics.

Visuo-haptic object perception for robots: an overview

The object perception capabilities of humans are impressive, and this becomes even more evident when trying to develop solutions with a similar proficiency in autonomous robots. While there have been notable advancements in the technologies for artificial vision and touch, the effective integration of these two sensory modalities in robotic applications still needs to be improved, and several open challenges exist. Taking inspiration from how humans combine visual and haptic perception to perceive object properties and drive the execution of manual tasks, this article summarises the current state of the art of visuo-haptic object perception in robots. Firstly, the biological basis of human multimodal object perception is outlined. Then, the latest advances in sensing technologies and data collection strategies for robots are discussed. Next, an overview of the main computational techniques is presented, highlighting the main challenges of multimodal machine learning and presenting a few representative articles in the areas of robotic object recognition, peripersonal space representation and manipulation. Finally, informed by the latest advancements and open challenges, this article outlines promising new research directions.

Visual illusions in Parkinson's disease: an interview survey of symptomatology

BACKGROUND

Several types of visual illusions can occur in Parkinson's disease (PD). However, the prevalence and types of specific illusions experienced by patients with PD remain unclear. This study aimed to investigate the types of illusions.

METHODS

A questionnaire of visual illusions was developed through a literature review in consultation with clinicians and neurologists. Based on the questionnaire, 40 consecutive patients with PD were asked a series of Yes/No questions regarding 20 types of visual illusions since the onset of PD. If participants answered 'Yes', they were then asked to detail their experience(s).

RESULTS

In total, 30 patients with PD had experienced visual illusions since disease onset; among them, 25 were still experiencing them at the time of the study. The most commonly observed illusion types were dysmorphopsia, complex visual illusions, metachromatopsia, and diplopia. Other observed illusions included textural illusions, macropsia, micropsia, teleopsia, pelopsia, kinetopsia, akinetopsia, Zeitraffer/Zeitlupen phenomena, tilt illusion, upside-down illusion, and palinopsia. Additionally, aberrant perception of surface orientation (inclination) was reported, which is yet to be reported in association with any disease. Visual illusions had detrimental effects on the patients' daily lives in some cases.

CONCLUSIONS

Systematic interviews regarding the incidence and details of visual illusions experienced by patients with PD could offer important information regarding their quality of life.

Tactile disgust: Post-contact can be more disgusting than contact

Several studies have examined if disgust can be evoked by contacting an object-yet none have examined if reported disgust changes when the hand leaves the object. This is surprising given that post-contact tactile disgust is probably a driver of hand hygiene. We examined contact and post-contact tactile disgust and its sensory origins. Participants were asked to touch several objects, making sensory, disgust, and desire-to-handwash evaluations. These ratings were made at three stages-of-contact: object-contact (just touch), post-contact (just touch), and visual post-contact (touch, vision). Disgust was typically highest at post-contact (when the hand left the object). Stickiness and wetness were uniquely predictive of object-contact disgust. Only stickiness drove post-contact disgust, and only wetness visual post-contact disgust. Hand-washing desire was primarily driven by quantity of residue perceived on the hand. These findings suggest that tactile disgust is a multisensory and iterative process relating to object- and residue-adhesiveness.

Dynamic dot displays reveal material motion network in the human brain

There is growing research interest in the neural mechanisms underlying the recognition of material categories and properties. This research field, however, is relatively more recent and limited compared to investigations of the neural mechanisms underlying object and scene category recognition. Motion is particularly important for the perception of non-rigid materials, but the neural basis of non-rigid material motion remains unexplored. Using fMRI, we investigated which brain regions respond preferentially to material motion versus other types of motion. We introduce a new database of stimuli - dynamic dot materials - that are animations of moving dots that induce vivid percepts of various materials in motion, e.g. flapping cloth, liquid waves, wobbling jelly. Control stimuli were scrambled versions of these same animations and rigid three-dimensional rotating dots. Results showed that isolating material motion properties with dynamic dots (in contrast with other kinds of motion) activates a network of cortical regions in both ventral and dorsal visual pathways, including areas normally associated with the processing of surface properties and shape, and extending to somatosensory and premotor cortices. We suggest that such a widespread preference for material motion is due to strong associations between stimulus properties. For example viewing dots moving in a specific pattern not only elicits percepts of material motion; one perceives a flexible, non-rigid shape, identifies the object as a cloth flapping in the wind, infers the object's weight under gravity, and anticipates how it would feel to reach out and touch the material. These results are a first important step in mapping out the cortical architecture and dynamics in material-related motion processing.

Shitsukan - the Multisensory Perception of Quality

We often estimate, or perceive, the quality of materials, surfaces, and objects, what the Japanese refer to as 'shitsukan', by means of several of our senses. The majority of the literature on shitsukan perception has, though, tended to focus on the unimodal visual evaluation of stimulus properties. In part, this presumably reflects the widespread hegemony of the visual in the modern era and, in part, is a result of the growing interest, not to mention the impressive advances, in digital rendering amongst the computer graphics community. Nevertheless, regardless of such an oculocentric bias in so much of the empirical literature, it is important to note that several other senses often do contribute to the impression of the material quality of surfaces, materials, and objects as experienced in the real world, rather than just in virtual reality. Understanding the multisensory contributions to the perception of material quality, especially when combined with computational and neural data, is likely to have implications for a number of fields of basic research as well as being applicable to emerging domains such as, for example, multisensory augmented retail, not to mention multisensory packaging design.

A meta-analysis of the size-weight and material-weight illusions

The current study comprises the first systematic meta-analysis of weight illusions. We obtained descriptive data from studies in which subjective heaviness estimates were made for pairs or groups of objects that had the same mass and different volumes (size-weight illusion; SWI) or different apparent material properties (material-weight illusion; MWI). Using these data, we calculated mean effect sizes to represent illusion strength. Other study details, including stimulus mass, volume, density, and degree of visual and somatosensory access to the stimuli were also recorded to quantify the contribution of these variables to effect sizes for the SWI. The results indicate that the SWI has a larger mean effect size than the MWI and that the former is consistent in strength when information about stimulus size is gained through somatosensory channels, regardless of visual access. The SWI is weaker when only the visual system provides size information. Effect sizes for the SWI were larger when there was a greater difference in volume across the stimuli. There was also a positive correlation between SWI strength and the difference in physical density across the different experimental stimuli, even after controlling for volume differences. Together, we argue that these findings provide support for theories of weight illusions that are based on conceptual expectancies as well as those that are based on bottom-up processing of physical density. We further propose that these processes, which have been considered dichotomously in the past, may not be mutually exclusive from each other and could both contribute to our perception of weight when we handle objects in everyday life.

Cortical Regions Encoding Hardness Perception Modulated by Visual Information Identified by Functional Magnetic Resonance Imaging With Multivoxel Pattern Analysis

Recent studies have revealed that hardness perception is determined by visual information along with the haptic input. This study investigated the cortical regions involved in hardness perception modulated by visual information using functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA). Twenty-two healthy participants were enrolled. They were required to place their left and right hands at the front and back, respectively, of a mirror attached to a platform placed above them while lying in a magnetic resonance scanner. In conditions SFT, MED, and HRD, one of three polyurethane foam pads of varying hardness (soft, medium, and hard, respectively) was presented to the left hand in a given trial, while only the medium pad was presented to the right hand in all trials. MED was defined as the control condition, because the visual and haptic information was congruent. During the scan, the participants were required to push the pad with the both hands while observing the reflection of the left hand and estimate the hardness of the pad perceived by the right (hidden) hand based on magnitude estimation. Behavioral results showed that the perceived hardness was significantly biased toward softer or harder in >73% of the trials in conditions SFT and HRD; we designated these trials as visually modulated (SFTvm and HRDvm, respectively). The accuracy map was calculated individually for each of the pair-wise comparisons of (SFTvm vs. MED), (HRDvm vs. MED), and (SFTvm vs. HRDvm) by a searchlight MVPA, and the cortical regions encoding the perceived hardness with visual modulation were identified by conjunction of the three accuracy maps in group analysis. The cluster was observed in the right sensory motor cortex, left anterior intraparietal sulcus (aIPS), bilateral parietal operculum (PO), and occipito-temporal cortex (OTC). Together with previous findings on such cortical regions, we conclude that the visual information of finger movements processed in the OTC may be integrated with haptic input in the left aIPS, and the subjective hardness perceived by the right hand with visual modulation may be processed in the cortical network between the left PO and aIPS.

Neural correlates of top-down modulation of haptic shape versus roughness perception

Exploring an object's shape by touch also renders information about its surface roughness. It has been suggested that shape and roughness are processed distinctly in the brain, a result based on comparing brain activation when exploring objects that differed in one of these features. To investigate the neural mechanisms of top‐down control on haptic perception of shape and roughness, we presented the same multidimensional objects but varied the relevance of each feature. Specifically, participants explored two objects that varied in shape (oblongness of cuboids) and surface roughness. They either had to compare the shape or the roughness in an alternative‐forced‐choice‐task. Moreover, we examined whether the activation strength of the identified brain regions as measured by functional magnetic resonance imaging (fMRI) can predict the behavioral performance in the haptic discrimination task. We observed a widespread network of activation for shape and roughness perception comprising bilateral precentral and postcentral gyrus, cerebellum, and insula. Task‐relevance of the object's shape increased activation in the right supramarginal gyrus (SMG/BA 40) and the right precentral gyrus (PreCG/BA 44) suggesting that activation in these areas does not merely reflect stimulus‐driven processes, such as exploring shape, but also entails top‐down controlled processes driven by task‐relevance. Moreover, the strength of the SMG/PreCG activation predicted individual performance in the shape but not in the roughness discrimination task. No activation was found for the reversed contrast (roughness > shape). We conclude that macrogeometric properties, such as shape, can be modulated by top‐down mechanisms whereas roughness, a microgeometric feature, seems to be processed automatically.

Neural Mechanisms of Material Perception: Quest on Shitsukan

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

Tactile-to-Visual Cross-Modal Transfer of Texture Categorisation Following Training: An fMRI Study

We investigated the neural underpinnings of texture categorisation using exemplars that were previously learned either within modalities (visual training and visual test) or across modalities (tactile training and visual test). Previous models of learning suggest a decrease in activation in brain regions that are typically involved in cognitive control during task acquisition, but a concomitant increase in activation in brain regions associated with the representation of the acquired information. In our study, participants were required to learn to categorise fabrics of different textures as either natural or synthetic. Training occurred over several sessions, with each fabric presented either visually or through touch to a participant. Pre- and post-training tests, in which participants categorised visual images only of the fabrics, were conducted during a functional magnetic resonance imaging (fMRI) scan. Consistent with previous research on cognitive processes involved in task acquisition, we found that categorisation training was associated with a decrease in activation in brain regions associated with cognitive systems involved in learning, including the superior parietal cortex, dorsal anterior cingulate cortex (dACC), and the right dorsolateral prefrontal cortex (DLFC). Moreover, these decreases were independent of training modality. In contrast, we found greater activation to visual textures in a region within the left medial occipital cortex (MOC) following training. There was no overall evidence of an effect of training modality in the main analyses, with texture-specific regional changes associated with both within- (visual) and cross- (touch) modal training. However, further analyses suggested that, unlike categorisation performance following within-modal training, crossmodal training was associated with bilateral activation of the MOC. Our results support previous evidence for a multisensory representation of texture within early visual regions of the cortex and provide insight into how multisensory categories are formed in the brain.

Evaluating Integration Strategies for Visuo-Haptic Object Recognition

In computational systems for visuo-haptic object recognition, vision and haptics are often modeled as separate processes. But this is far from what really happens in the human brain, where cross- as well as multimodal interactions take place between the two sensory modalities. Generally, three main principles can be identified as underlying the processing of the visual and haptic object-related stimuli in the brain: (1) hierarchical processing, (2) the divergence of the processing onto substreams for object shape and material perception, and (3) the experience-driven self-organization of the integratory neural circuits. The question arises whether an object recognition system can benefit in terms of performance from adopting these brain-inspired processing principles for the integration of the visual and haptic inputs. To address this, we compare the integration strategy that incorporates all three principles to the two commonly used integration strategies in the literature. We collected data with a NAO robot enhanced with inexpensive contact microphones as tactile sensors. The results of our experiments involving every-day objects indicate that (1) the contact microphones are a good alternative to capturing tactile information and that (2) organizing the processing of the visual and haptic inputs hierarchically and in two pre-processing streams is helpful performance-wise. Nevertheless, further research is needed to effectively quantify the role of each identified principle by itself as well as in combination with others.

Analysis of haptic information in the cerebral cortex

Haptic sensing of objects acquires information about a number of properties. This review summarizes current understanding about how these properties are processed in the cerebral cortex of macaques and humans. Nonnoxious somatosensory inputs, after initial processing in primary somatosensory cortex, are partially segregated into different pathways. A ventrally directed pathway carries information about surface texture into parietal opercular cortex and thence to medial occipital cortex. A dorsally directed pathway transmits information regarding the location of features on objects to the intraparietal sulcus and frontal eye fields. Shape processing occurs mainly in the intraparietal sulcus and lateral occipital complex, while orientation processing is distributed across primary somatosensory cortex, the parietal operculum, the anterior intraparietal sulcus, and a parieto-occipital region. For each of these properties, the respective areas outside primary somatosensory cortex also process corresponding visual information and are thus multisensory. Consistent with the distributed neural processing of haptic object properties, tactile spatial acuity depends on interaction between bottom-up tactile inputs and top-down attentional signals in a distributed neural network. Future work should clarify the roles of the various brain regions and how they interact at the network level.

When Neuroscience 'Touches' Architecture: From Hapticity to a Supramodal Functioning of the Human Brain

In the last decades, the rapid growth of functional brain imaging methodologies allowed cognitive neuroscience to address open questions in philosophy and social sciences. At the same time, novel insights from cognitive neuroscience research have begun to influence various disciplines, leading to a turn to cognition and emotion in the fields of planning and architectural design. Since 2003, the Academy of Neuroscience for Architecture has been supporting 'neuro-architecture' as a way to connect neuroscience and the study of behavioral responses to the built environment. Among the many topics related to multisensory perceptual integration and embodiment, the concept of hapticity was recently introduced, suggesting a pivotal role of tactile perception and haptic imagery in architectural appraisal. Arguments have thus risen in favor of the existence of shared cognitive foundations between hapticity and the supramodal functional architecture of the human brain. Precisely, supramodality refers to the functional feature of defined brain regions to process and represent specific information content in a more abstract way, independently of the sensory modality conveying such information to the brain. Here, we highlight some commonalities and differences between the concepts of hapticity and supramodality according to the distinctive perspectives of architecture and cognitive neuroscience. This comparison and connection between these two different approaches may lead to novel observations in regard to people-environment relationships, and even provide empirical foundations for a renewed evidence-based design theory.

Associated and dissociated neural substrates of aesthetic judgment and aesthetic emotion during the appreciation of everyday designed products

The aesthetics of designed products have become part of our life in modern society. This study explores the neural mechanisms of how aesthetic judgment and aesthetic emotion interplay during the appreciation of designed products that are commonly seen in daily life. Participants were 30 college students, and the stimuli were 90 pictures of everyday designed products. Based on an event-related paradigm, the findings of this study suggest that there are associative and dissociative neutral mechanisms underlying different types of aesthetic judgment and aesthetic emotion. The study identified the following main findings: (a) normative beauty and subjective beauty both involved the left anterior cingulate cortex (ACC); (b) subjective beauty and positive emotion both involved the right ACC; (c) subjective beauty and negative emotion both involved the precuneus; (d) subjective ugliness and negative emotion both involved the right inferior frontal gyrus; (e) subjective ugliness alone additionally activated the insula; and (f) subjective beauty alone additionally activated the caudate. The findings in this study shed light on complex but ordinary processes of aesthetic appreciation.

The embodied dynamics of perceptual causality: a slippery slope?

In Michotte's launching displays, while the launcher (object A) seems to move autonomously, the target (object B) seems to be displaced passively. However, the impression of A actively launching B does not persist beyond a certain distance identified as the "radius of action" of A over B. If the target keeps moving beyond the radius of action, it loses its passivity and seems to move autonomously. Here, we manipulated implied friction by drawing (or not) a surface upon which A and B are traveling, and by varying the inclination of this surface in screen- and earth-centered reference frames. Among 72 participants (n = 52 in Experiment 1; n = 20 in Experiment 2), we show that both physical embodiment of the event (looking straight ahead at a screen displaying the event on a vertical plane vs. looking downwards at the event displayed on a horizontal plane) and contextual information (objects moving along a depicted surface or in isolation) affect interpretation of the event and modulate the radius of action of the launcher. Using classical mechanics equations, we show that representational consistency of friction from radius of action responses emphasizes the embodied nature of frictional force in our cognitive architecture.

The Effect of Task Instruction on Haptic Texture Processing: The Neural Underpinning of Roughness and Spatial Density Perception

Perceived roughness is associated with a variety of physical factors and multiple peripheral afferent types. The current study investigated whether this complexity of the mapping between physical and perceptual space is reflected at the cortical level. In an integrative psychophysical and imaging approach, we used dot pattern stimuli for which previous studies reported a simple linear relationship of interdot spacing and perceived spatial density and a more complex function of perceived roughness. Thus, by using both a roughness and a spatial estimation task, the physical and perceived stimulus characteristics could be dissociated, with the spatial density task controlling for the processing of low-level sensory aspects. Multivoxel pattern analysis was used to investigate which brain regions hold information indicative of the level of the perceived texture characteristics. While information about differences in perceived roughness was primarily available in higher-order cortices, that is, the operculo-insular cortex and a ventral visual cortex region, information about perceived spatial density could already be derived from early somatosensory and visual regions. This result indicates that cortical processing reflects the different complexities of the evaluated haptic texture dimensions. Furthermore, this study is to our knowledge the first to show a contribution of the visual cortex to tactile roughness perception.

Parahippocampal cortex is involved in material processing via echoes in blind echolocation experts

Some blind humans use sound to navigate by emitting mouth-clicks and listening to the echoes that reflect from silent objects and surfaces in their surroundings. These echoes contain information about the size, shape, location, and material properties of objects. Here we present results from an fMRI experiment that investigated the neural activity underlying the processing of materials through echolocation. Three blind echolocation experts (as well as three blind and three sighted non-echolocating control participants) took part in the experiment. First, we made binaural sound recordings in the ears of each echolocator while he produced clicks in the presence of one of three different materials (fleece, synthetic foliage, or whiteboard), or while he made clicks in an empty room. During fMRI scanning these recordings were played back to participants. Remarkably, all participants were able to identify each of the three materials reliably, as well as the empty room. Furthermore, a whole brain analysis, in which we isolated the processing of just the reflected echoes, revealed a material-related increase in BOLD activation in a region of left parahippocampal cortex in the echolocating participants, but not in the blind or sighted control participants. Our results, in combination with previous findings about brain areas involved in material processing, are consistent with the idea that material processing by means of echolocation relies on a multi-modal material processing area in parahippocampal cortex.