Sensory Substitution and Multimodal Mental Imagery

Exploring Group Differences in the Crossmodal Correspondences

There has been a rapid growth of interest amongst researchers in the cross-modal correspondences in recent years. In part, this has resulted from the emerging realization of the important role that the correspondences can sometimes play in multisensory integration. In turn, this has led to an interest in the nature of any differences between individuals, or rather, between groups of individuals, in the strength and/or consensuality of cross-modal correspondences that may be observed in both neurotypically normal groups cross-culturally, developmentally, and across various special populations (including those who have lost a sense, as well as those with autistic tendencies). The hope is that our emerging understanding of such group differences may one day provide grounds for supporting the reality of the various different types of correspondence that have so far been proposed, namely structural, statistical, semantic, and hedonic (or emotionally mediated).

Sensory Perception Mechanism for Preparing the Combinations of Stimuli Operation in the Architectural Experience

Sensory stimuli in an architectural space play an important role in the human perception of the indoor environment, no matter whether they are static or dynamic, isolated, or combined. By enhancing some perceptions in the sensory stimuli, the overall perceptions of an architectural space can be improved, especially for an intelligent architectural space. As yet, there are few studies reported about the sensory perception mechanism for the sensory stimuli operation in the architectural experience. In this research, a wooden micro building was prepared for the study of the sensitivity level of participants to various sensory stimuli in the same and in different sensory domains. Participants’ visual, auditory, olfactory, tactile and kinaesthesia perceptions were discussed statistically in terms of the sensitivity level. Based on the study, the effect of a single dynamic sensory stimulus (a dynamically coloured light) on the participants’ perception was studied in a paper architectural model from two aspects including preference and emotion. The dynamically coloured light was discussed statistically in terms of the level of preference. The study showed that there are significant differences among participants’ levels of sensitivity to the different sensory domains and to the different sensory stimuli. In particular, the sensitivity level to the stimulus that is the colour of a space is the highest of all stimuli. As a single changing sensory stimulus, a dynamically coloured light can lead to significant mood fluctuations and changes in the preference level. In particular, yellow is the favourite colour of light. The object of this study is expected to provide a theoretical foundation that is related to sensory choice, sensory perception enhancement and the combination forms of sensory perceptions. Based on the theoretical foundation, the perception design of overlapped multi-sensory stimuli and a single dynamic stimulus can be conducted to improve the quality of the indoor environment of normal and intelligent multi-sensory architecture.

Amodal completion and relationalism

Amodal completion is usually characterized as the representation of those parts of the perceived object that we get no sensory stimulation from. In the case of the visual sense modality, for example, amodal completion is the representation of occluded parts of objects we see. I argue that relationalism about perception, the view that perceptual experience is constituted by the relation to the perceived object, cannot give a coherent account of amodal completion. The relationalist has two options: construe the perceptual relation as the relation to the entire perceived object or as the relation to the unoccluded parts of the perceived object. I argue that neither of these options are viable.

Augmenting perception: How artificial intelligence transforms sensory substitution

What happens when artificial sensors are coupled with the human senses? Using technology to extend the senses is an old human dream, on which sensory substitution and other augmentation technologies have already delivered. Laser tactile canes, corneal implants and magnetic belts can correct or extend what individuals could otherwise perceive. Here we show why accommodating intelligent sensory augmentation devices not just improves but also changes the way of thinking and classifying former sensory augmentation devices. We review the benefits in terms of signal processing and show why non-linear transformation is more than a mere improvement compared to classical linear transformation.

Harnessing emotional mental imagery to reduce anxiety and depression in young people: an integrative review of progress and promise

Emotional mental imagery is a powerful part of our mental landscape. Given its capacity to depict, process, and generate emotional events, mental imagery could have an important role in psychological therapies. This Series paper explores whether harnessing emotional mental imagery is meaningful to young people; ways in which interventions use emotional mental imagery; contextual and individual factors influencing intervention effectiveness; and mechanisms underpinning imagery techniques. We completed a systematic review of imagery interventions and consulted young people with lived experience (n=10) and leading international experts (n=7). The systematic search identified 86 papers covering a diverse range of imagery interventions. Across the seven categories of techniques reviewed, imagery rescripting for aversive memories, techniques targeting positive imagery, and imagery-enhanced protocols indicated the most potential. The report suggests that harnessing emotional mental imagery in psychological interventions could be a promising approach to reduce anxiety and depression and that mental health science could inform the development of new interventions and help to maximise intervention effectiveness.

Training-induced plasticity enables visualizing sounds with a visual-to-auditory conversion device

Sensory substitution devices aim at restoring visual functions by converting visual information into auditory or tactile stimuli. Although these devices show promise in the range of behavioral abilities they allow, the processes underlying their use remain underspecified. In particular, while an initial debate focused on the visual versus auditory or tactile nature of sensory substitution, since over a decade, the idea that it reflects a mixture of both has emerged. In order to investigate behaviorally the extent to which visual and auditory processes are involved, participants completed a Stroop-like crossmodal interference paradigm before and after being trained with a conversion device which translates visual images into sounds. In addition, participants' auditory abilities and their phenomenologies were measured. Our study revealed that, after training, when asked to identify sounds, processes shared with vision were involved, as participants’ performance in sound identification was influenced by the simultaneously presented visual distractors. In addition, participants’ performance during training and their associated phenomenology depended on their auditory abilities, revealing that processing finds its roots in the input sensory modality. Our results pave the way for improving the design and learning of these devices by taking into account inter-individual differences in auditory and visual perceptual strategies.

The Virtual Challenge: Virtual Reality Tools for Intervention in Children with Developmental Coordination Disorder

This narrative review highlights the latest achievements in the field of tele-rehabilitation: Virtual Reality (VR) and Augmented Reality (AR) serious games aimed at restoring and improving cognitive functions could be effectively used in Developmental Coordination Disorder Training. Studies investigating the effects of the abovementioned tech applications on cognitive improvement have been considered, following a comprehensive literature search in the scientific electronic databases: Pubmed, Scopus, Plos One, ScienceDirect. This review investigates the effects of VR and AR in improving space/motor skills through mental images manipulation training in children with developmental coordination disorders. The results revealed that in spite of the spreading of technology, actually only four studies investigated the effects of VR/AR tools on mental images manipulation. This study highlights new, promising VR and AR based therapeutic opportunities for digital natives now available, emphasizing the advantages of using motivational reward-oriented tools, in a playful therapeutic environment. However, more research in this filed is needed to identify the most effective virtual tool set for clinical use.

Offline perception: an introduction

Experiences that are self-generated and independent of sensory stimulations permeate our whole life. This theme issue examines their similarities and differences, systematizes the literature from an integrative perspective, critically discusses state-of-the-art empirical findings and proposes new theoretical approaches. The aim of the theme issue is to foster interaction between the different disciplines and research directions involved and to explore the prospects of a unificatory account of offline perception in general. This article is part of the theme issue 'Offline perception: voluntary and spontaneous perceptual experiences without matching external stimulation'.

Synesthesia as (Multimodal) Mental Imagery

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one's mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory stimulation.

Cross-modal size-contrast illusion: Acoustic increases in intensity and bandwidth modulate haptic representation of object size

Changes in the retinal size of stationary objects provide a cue to the observer's motion in the environment: Increases indicate the observer's forward motion, and decreases backward motion. In this study, a series of images each comprising a pair of pine-tree figures were translated into auditory modality using sensory substitution software. Resulting auditory stimuli were presented in an ascending sequence (i.e. increasing in intensity and bandwidth compatible with forward motion), descending sequence (i.e. decreasing in intensity and bandwidth compatible with backward motion), or in a scrambled order. During the presentation of stimuli, blindfolded participants estimated the lengths of wooden sticks by haptics. Results showed that those exposed to the stimuli compatible with forward motion underestimated the lengths of the sticks. This consistent underestimation may share some aspects with visual size-contrast effects such as the Ebbinghaus illusion. In contrast, participants in the other two conditions did not show such magnitude of error in size estimation; which is consistent with the "adaptive perceptual bias" towards acoustic increases in intensity and bandwidth. In sum, we report a novel cross-modal size-contrast illusion, which reveals that auditory motion cues compatible with listeners' forward motion modulate haptic representations of object size.

Perception is not all-purpose

I aim to show that perception depends counterfactually on the action we want to perform. Perception is not all-purpose: what we want to do does influence what we see. After clarifying how this claim is different from the one at stake in the cognitive penetrability debate and what counterfactual dependence means in my claim, I will give a two-step argument: (a) one's perceptual attention depends counterfactually on one's intention to perform an action (everything else being equal) and (b) one's perceptual processing depends counterfactually on one's perceptual attention (everything else being equal). If we put these claims together, what we get is that one's perceptual processing depends counterfactually on one's intention to perform an action (everything else being equal).

Multimodal mental imagery

When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception – I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality, for example, when I hear the noisy coffee machine in the next room, that is, without seeing it. The aim of this paper is to bring together empirical findings about multimodal perception and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like this, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition). Multimodal mental imagery is not a rare and obscure phenomenon. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality – like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved – all the other aspects of these multisensory events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception.