Tactile exploration of virtual objects for blind and sighted people: the role of beta 1 EEG band in sensory substitution and supramodal mental mapping

EEG-based functional connectivity for tactile roughness discrimination

Tactile sensation and perception involve cooperation between different parts of the brain. Roughness discrimination is an important phase of texture recognition. In this study, we investigated how different roughness levels would influence the brain network characteristics. We recorded EEG signals from nine right-handed healthy subjects who underwent touching three surfaces with different levels of roughness. The experiment was separately repeated in 108 trials for each hand for both static and dynamic touch. For estimation of the functional connectivity between brain regions, the phase lag index method was employed. Frequency-specific connectivity patterns were observed in the ipsilateral and contralateral hemispheres to the hand of interest, for delta, theta, alpha, and beta frequency bands under the study. A number of connections were identified to be in charge of discrimination between surfaces in both alpha and beta frequency bands for the left hand in static touch and for the right hand in dynamic touch. In addition, common connections were determined in both hands for all three roughness in alpha band for static touch and in theta band for dynamic touch. The common connections were identified for the smooth surface in beta band for static touch and in delta and alpha bands for dynamic touch. As observed for static touch in alpha band and for dynamic touch in theta band, the number of common connections between the two hands was decreased by increasing the surface roughness. The results of this research would extend the current knowledge about tactile information processing in the brain.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-022-09876-1.

Spatial Cognition of the Visually Impaired: A Case Study in a Familiar Environment

OBJECTIVES

This paper aims to explore the factors influencing the spatial cognition of the visually impaired in familiar environments.

BACKGROUND

Massage hospitals are some of the few places that can provide work for the visually impaired in China. Studying the spatial cognition of the visually impaired in a massage hospital could be instructive for the design of working environments for the visually impaired and other workplaces in the future.

METHODS

First, the subjective spatial cognition of the visually impaired was evaluated by object layout tasks for describing the spatial relationships among object parts. Second, physiological monitoring signal data, including the electrodermal activity, heart rate variability, and electroencephalography, were collected while the visually impaired doctors walked along prescribed routes based on the feature analysis of the physical environment in the hospital, and then their physiological monitoring signal data for each route were compared. The visual factors, physical environmental factors, and human-environment interactive factors that significantly impact the spatial cognition of visually impaired people were discussed.

CONCLUSIONS

(1) visual acuity affects the spatial cognition of the visually impaired in familiar environments; (2) the spatial cognition of the visually impaired can be promoted by a longer staying time and the more regular sequence of a physical environment; (3) the spatial comfort of the visually impaired can be improved by increasing the amount of greenery; and (4) the visual comfort of the visually impaired can be reduced by rich interior colors and contrasting lattice floor tiles.

The reality of virtual reality

Virtual reality (VR) has become a popular tool for investigating human behavior and brain functions. Nevertheless, it is unclear whether VR constitutes an actual form of reality or is more like an advanced simulation. Determining the nature of VR has been mostly achieved by self-reported presence measurements, defined as the feeling of being submerged in the experience. However, subjective measurements might be prone to bias and, most importantly, do not allow for a comparison with real-life experiences. Here, we show that real-life and VR height exposures using 3D-360° videos are mostly indistinguishable on a psychophysiological level (EEG and HRV), while both differ from a conventional 2D laboratory setting. Using a fire truck, three groups of participants experienced a real-life (N = 25), a virtual (N = 24), or a 2D laboratory (N = 25) height exposure. Behavioral and psychophysiological results suggest that identical exogenous and endogenous cognitive as well as emotional mechanisms are deployed to process the real-life and virtual experience. Specifically, alpha- and theta-band oscillations in line with heart rate variability, indexing vigilance, and anxiety were barely indistinguishable between those two conditions, while they differed significantly from the laboratory setup. Sensory processing, as reflected by beta-band oscillations, exhibits a different pattern for all conditions, indicating further room for improving VR on a haptic level. In conclusion, the study shows that contemporary photorealistic VR setups are technologically capable of mimicking reality, thus paving the way for the investigation of real-world cognitive and emotional processes under controlled laboratory conditions. For a video summary, see https://youtu.be/fPIrIajpfiA.

Cognitive map formation supported by auditory, haptic, and multimodal information in persons with blindness

For efficient navigation, the brain needs to adequately represent the environment in a cognitive map. In this review, we sought to give an overview of literature about cognitive map formation based on non-visual modalities in persons with blindness (PWBs) and sighted persons. The review is focused on the auditory and haptic modalities, including research that combines multiple modalities and real-world navigation. Furthermore, we addressed implications of route and survey representations. Taking together, PWBs as well as sighted persons can build up cognitive maps based on non-visual modalities, although the accuracy sometime somewhat differs between PWBs and sighted persons. We provide some speculations on how to deploy information from different modalities to support cognitive map formation. Furthermore, PWBs and sighted persons seem to be able to construct route as well as survey representations. PWBs can experience difficulties building up a survey representation, but this is not always the case, and research suggests that they can acquire this ability with sufficient spatial information or training. We discuss possible explanations of these inconsistencies.

Enabling visually impaired people to learn three-dimensional tactile graphics with a 3DOF haptic mouse

Background

In this work, we present a novel sensory substitution system that enables to learn three dimensional digital information via touch when vision is unavailable. The system is based on a mouse-shaped device, designed to jointly perceive, with one finger only, local tactile height and inclination cues of arbitrary scalar fields. The device hosts a tactile actuator with three degrees of freedom: elevation, roll and pitch. The actuator approximates the tactile interaction with a plane tangential to the contact point between the finger and the field. Spatial information can therefore be mentally constructed by integrating local and global tactile cues: the actuator provides local cues, whereas proprioception associated with the mouse motion provides the global cues.

Methods

The efficacy of the system is measured by a virtual/real object-matching task. Twenty-four gender and age-matched participants (one blind and one blindfolded sighted group) matched a tactile dictionary of virtual objects with their 3D-printed solid version. The exploration of the virtual objects happened in three conditions, i.e., with isolated or combined height and inclination cues. We investigated the performance and the mental cost of approximating virtual objects in these tactile conditions.

Results

In both groups, elevation and inclination cues were sufficient to recognize the tactile dictionary, but their combination worked at best. The presence of elevation decreased a subjective estimate of mental effort. Interestingly, only visually impaired participants were aware of their performance and were able to predict it.

Conclusions

The proposed technology could facilitate the learning of science, engineering and mathematics in absence of vision, being also an industrial low-cost solution to make graphical user interfaces accessible for people with vision loss.

The effectiveness of an interactive audio-tactile map for the process of cognitive mapping and recall among people with visual impairments

BACKGROUND

People with visual impairments can experience numerous challenges navigating unfamiliar environments. Systems that operate as prenavigation tools can assist such individuals. This mixed-methods study examined the effectiveness of an interactive audio-tactile map tool on the process of cognitive mapping and recall, among people who were blind or had visual impairments. The tool was developed with the involvement of visually impaired individuals who additionally provided further feedback throughout this research.

METHODS

A mixed-methods experimental design was employed. Fourteen participants were allocated to either an experimental group who were exposed to an audio-tactile map, or a control group exposed to a verbally annotated tactile map. After five minutes' exposure, multiple-choice questions examined participants' recall of the spatial and navigational content. Subsequent semi-structured interviews were conducted to examine their views surrounding the study and the product.

RESULTS

The experimental condition had significantly better overall recall than the control group and higher average scores in all four areas examined by the questions. The interviews suggested that the interactive component offered individuals the freedom to learn the map in several ways and did not restrict them to a sequential and linear approach to learning.

CONCLUSION

Assistive technology can reduce challenges faced by people with visual impairments, and the flexible learning approach offered by the audio-tactile map may be of particular value. Future researchers and assistive technology developers may wish to explore this further.

Improving spatial working memory in blind and sighted youngsters using programmable tactile displays

OBJECTIVE

To investigate whether training with tactile matrices displayed with a programmable tactile display improves recalling performance of spatial images in blind, low-vision and sighted youngsters. To code and understand the behavioral underpinnings of learning two-dimensional tactile dispositions, in terms of spontaneous exploration strategies.

METHODS

Three groups of blind, low-vision and sighted youngsters between 6 and 18 years old performed four training sessions with a weekly schedule in which they were asked to memorize single or double spatial layouts, featured as two-dimensional matrices.

RESULTS

Results showed that all groups of participants significantly improved their recall performance compared to the first session baseline in the single-matrix task. No statistical difference in performance between groups emerged in this task. Instead, the learning effect in visually impaired participants is reduced in the double-matrix task, whereas it is still robust in blindfolded sighted controls. We also coded tactile exploration strategies in both tasks and their correlation with performance. Sighted youngsters, in particular, favored a proprioceptive exploration strategy. Finally, performance in the double-matrix task negatively correlated with using one hand and positively correlated with a proprioceptive strategy.

CONCLUSION

The results of our study indicate that blind persons do not easily process two separate spatial layouts. However, rehabilitation programs promoting bi-manual and proprioceptive approaches to tactile exploration might help improve spatial abilities. Finally, programmable tactile displays are an effective way to make spatial and graphical configurations accessible to visually impaired youngsters and they can be profitably exploited in rehabilitation.

Hearing Shapes: Event-related Potentials Reveal the Time Course of Auditory-Visual Sensory Substitution

In auditory-visual sensory substitution, visual information (e.g., shape) can be extracted through strictly auditory input (e.g., soundscapes). Previous studies have shown that image-to-sound conversions that follow simple rules [such as the Meijer algorithm; Meijer, P. B. L. An experimental system for auditory image representation. Transactions on Biomedical Engineering, 39, 111-121, 1992] are highly intuitive and rapidly learned by both blind and sighted individuals. A number of recent fMRI studies have begun to explore the neuroplastic changes that result from sensory substitution training. However, the time course of cross-sensory information transfer in sensory substitution is largely unexplored and may offer insights into the underlying neural mechanisms. In this study, we recorded ERPs to soundscapes before and after sighted participants were trained with the Meijer algorithm. We compared these posttraining versus pretraining ERP differences with those of a control group who received the same set of 80 auditory/visual stimuli but with arbitrary pairings during training. Our behavioral results confirmed the rapid acquisition of cross-sensory mappings, and the group trained with the Meijer algorithm was able to generalize their learning to novel soundscapes at impressive levels of accuracy. The ERP results revealed an early cross-sensory learning effect (150-210 msec) that was significantly enhanced in the algorithm-trained group compared with the control group as well as a later difference (420-480 msec) that was unique to the algorithm-trained group. These ERP modulations are consistent with previous fMRI results and provide additional insight into the time course of cross-sensory information transfer in sensory substitution.

Effects of stimulus mode and ambient temperature on cerebral responses to local thermal stimulation: An EEG study

The physiological responses to human thermal stimulation have been widely investigated, but most of them are mainly concerned about the whole body thermal stimulation. In this study, we investigated the effects of stimulus mode and ambient temperature on cerebral responses during local thermal stimulation on hand. The left hands were stimulated by metal thermostat based and thermostatic water based stimulators at different stimulated temperatures (38°C, 40°C, 42°C and 44°C) and different ambient temperatures (25°C and 32°C). EEG data were recorded over the whole brain during the experiments. Then the statistical comparisons were conducted on the EEG relative power among different experimental sessions. We observed that EEG activities were alternated between thermal stimulated periods and the baseline in all four frequency bands. And there was a higher percentage of delta band power in the right temporal and parietal regions under the ambient temperature of 32°C while compared to 25°C. In addition, the theta band activity under the metal based stimulation showed significantly higher EEG relative power than that under the water based stimulation over the whole brain. Compared with the water based stimulation, there was a lower EEG relative power of the beta band activity during the metal based stimulation in the bilateral frontal and right temporal regions. The experimental results suggested that the neural physiological responses in different EEG frequency bands were sensitive to different influence factors during the local hand thermal stimulation.

Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception

Theories of embodied cognition propose that perception is shaped by sensory stimuli and by the actions of the organism. Following sensorimotor contingency theory, the mastery of lawful relations between own behavior and resulting changes in sensory signals, called sensorimotor contingencies, is constitutive of conscious perception. Sensorimotor contingency theory predicts that, after training, knowledge relating to new sensorimotor contingencies develops, leading to changes in the activation of sensorimotor systems, and concomitant changes in perception. In the present study, we spell out this hypothesis in detail and investigate whether it is possible to learn new sensorimotor contingencies by sensory augmentation. Specifically, we designed an fMRI compatible sensory augmentation device, the feelSpace belt, which gives orientation information about the direction of magnetic north via vibrotactile stimulation on the waist of participants. In a longitudinal study, participants trained with this belt for seven weeks in natural environment. Our EEG results indicate that training with the belt leads to changes in sleep architecture early in the training phase, compatible with the consolidation of procedural learning as well as increased sensorimotor processing and motor programming. The fMRI results suggest that training entails activity in sensory as well as higher motor centers and brain areas known to be involved in navigation. These neural changes are accompanied with changes in how space and the belt signal are perceived, as well as with increased trust in navigational ability. Thus, our data on physiological processes and subjective experiences are compatible with the hypothesis that new sensorimotor contingencies can be acquired using sensory augmentation.

Tactile Perception for Stroke Induce Changes in Electroencephalography

Objective/Background

Tactile perception is a basic way to obtain and evaluate information about an object. The purpose of this study was to examine the effects of tactile perception on brain activation using two different tactile explorations, passive and active touches, in individuals with chronic hemiparetic stroke.

Methods

Twenty patients who were diagnosed with stroke (8 right brain damaged, 12 left brain damaged) participated in this study. The tactile perception was conducted using passive and active explorations in a sitting position. To determine the neurological changes in the brain, this study measured the brain waves of the participants using electroencephalography (EEG).

Results

The relative power of the sensory motor rhythm on the right prefrontal lobe and right parietal lobe was significantly greater during the active tactile exploration compared to the relative power during the passive exploration in the left damaged hemisphere. Most of the measured brain areas showed nonsignificantly higher relative power of the sensory motor rhythm during the active tactile exploration, regardless of which hemisphere was damaged.

Conclusion

The results of this study provided a neurophysiological evidence on tactile perception in individuals with chronic stroke. Occupational therapists should consider an active tactile exploration as a useful modality on occupational performance in rehabilitation training.

The Influence of Tactile Cognitive Maps on Auditory Space Perception in Sighted Persons

We have recently shown that vision is important to improve spatial auditory cognition. In this study, we investigate whether touch is as effective as vision to create a cognitive map of a soundscape. In particular, we tested whether the creation of a mental representation of a room, obtained through tactile exploration of a 3D model, can influence the perception of a complex auditory task in sighted people. We tested two groups of blindfolded sighted people – one experimental and one control group – in an auditory space bisection task. In the first group, the bisection task was performed three times: specifically, the participants explored with their hands the 3D tactile model of the room and were led along the perimeter of the room between the first and the second execution of the space bisection. Then, they were allowed to remove the blindfold for a few minutes and look at the room between the second and third execution of the space bisection. Instead, the control group repeated for two consecutive times the space bisection task without performing any environmental exploration in between. Considering the first execution as a baseline, we found an improvement in the precision after the tactile exploration of the 3D model. Interestingly, no additional gain was obtained when room observation followed the tactile exploration, suggesting that no additional gain was obtained by vision cues after spatial tactile cues were internalized. No improvement was found between the first and the second execution of the space bisection without environmental exploration in the control group, suggesting that the improvement was not due to task learning. Our results show that tactile information modulates the precision of an ongoing space auditory task as well as visual information. This suggests that cognitive maps elicited by touch may participate in cross-modal calibration and supra-modal representations of space that increase implicit knowledge about sound propagation.

The Effect of Programmable Tactile Displays on Spatial Learning Skills in Children and Adolescents of Different Visual Disability

Vision loss has severe impacts on physical, social and emotional well-being. The education of blind children poses issues as many scholar disciplines (e.g., geometry, mathematics) are normally taught by heavily relying on vision. Touch-based assistive technologies are potential tools to provide graphical contents to blind users, improving learning possibilities and social inclusion. Raised-lines drawings are still the golden standard, but stimuli cannot be reconfigured or adapted and the blind person constantly requires assistance. Although much research concerns technological development, little work concerned the assessment of programmable tactile graphics, in educative and rehabilitative contexts. Here we designed, on programmable tactile displays, tests aimed at assessing spatial memory skills and shapes recognition abilities. Tests involved a group of blind and a group of low vision children and adolescents in a four-week longitudinal schedule. After establishing subject-specific difficulty levels, we observed a significant enhancement of performance across sessions and for both groups. Learning effects were comparable to raised paper control tests: however, our setup required minimal external assistance. Overall, our results demonstrate that programmable maps are an effective way to display graphical contents in educative/rehabilitative contexts. They can be at least as effective as traditional paper tests yet providing superior flexibility and versatility.

The Importance of Visual Experience, Gender, and Emotion in the Assessment of an Assistive Tactile Mouse

Tactile maps are efficient tools to improve spatial understanding and mobility skills of visually impaired people. Their limited adaptability can be compensated with haptic devices which display graphical information, but their assessment is frequently limited to performance-based metrics only which can hide potential spatial abilities in O&M protocols. We assess a low-tech tactile mouse able to deliver three-dimensional content considering how performance, mental workload, behavior, and anxiety status vary with task difficulty and gender in congenitally blind, late blind, and sighted subjects. Results show that task difficulty coherently modulates the efficiency and difficulty to build mental maps, regardless of visual experience. Although exhibiting attitudes that were similar and gender-independent, the females had lower performance and higher cognitive load, especially when congenitally blind. All groups showed a significant decrease in anxiety after using the device. Tactile graphics with our device seems therefore to be applicable with different visual experiences, with no negative emotional consequences of mentally demanding spatial tasks. Going beyond performance-based assessment, our methodology can help with better targeting technological solutions in orientation and mobility protocols.

Predicting successful tactile mapping of virtual objects

Improving spatial ability of blind and visually impaired people is the main target of orientation and mobility (O&M) programs. In this study, we use a minimalistic mouse-shaped haptic device to show a new approach aimed at evaluating devices providing tactile representations of virtual objects. We consider psychophysical, behavioral, and subjective parameters to clarify under which circumstances mental representations of spaces (cognitive maps) can be efficiently constructed with touch by blindfolded sighted subjects. We study two complementary processes that determine map construction: low-level perception (in a passive stimulation task) and high-level information integration (in an active exploration task). We show that jointly considering a behavioral measure of information acquisition and a subjective measure of cognitive load can give an accurate prediction and a practical interpretation of mapping performance. Our simple TActile MOuse (TAMO) uses haptics to assess spatial ability: this may help individuals who are blind or visually impaired to be better evaluated by O&M practitioners or to evaluate their own performance.