The Perception of Space and Form Recognition in a Simulated Environment: The Case of Minimalist Sensory-Substitution Devices

Recurrence analysis of sensorimotor trajectories in a minimalist perceptual task using sonification

Active Perception perspectives claim that action is closely related to perception. An empirical approach that supports these theories is the minimalist, in which participants perform a task using an interface that provides minimal information. Their exploratory movements are crucial to generating a meaningful sequence of information. Previous studies analyzed sensorimotor trajectories describing qualitative strategies and linear quantification of participants' movement performance, but that approach struggles to capture the behavior of non-stationary data. In the present study, we applied the recurrence plot (RP) and recurrence quantification analysis (RQA) to study the structure of sensorimotor trajectories developed by participants trying to discriminate between two invisible geometric shapes (Triangle or Rectangle). The exploratory movements were made using a computer mouse and sonification-mediated feedback was provided, which depended exclusively on whether the pointer was inside or outside the shape. We applied RP and RQA to the sensorimotor trajectories, with the aim of studying their fine structure characteristics, focusing on their repetitive patterns. Recurrence analysis proved to be useful for quantifying differences in dynamic behavior that emerge when participants explore invisible virtual geometric shapes. The differences obtained in RQA-based measures associated with the vertical structures allowed to postulate the existence of particular exploration strategies for each figure. It was also possible to determine that the complexity of the dynamics changed according to the shape. We discuss these results in light of antecedents in haptic and visual perceptual exploration.

Training-induced plasticity enables visualizing sounds with a visual-to-auditory conversion device

Sensory substitution devices aim at restoring visual functions by converting visual information into auditory or tactile stimuli. Although these devices show promise in the range of behavioral abilities they allow, the processes underlying their use remain underspecified. In particular, while an initial debate focused on the visual versus auditory or tactile nature of sensory substitution, since over a decade, the idea that it reflects a mixture of both has emerged. In order to investigate behaviorally the extent to which visual and auditory processes are involved, participants completed a Stroop-like crossmodal interference paradigm before and after being trained with a conversion device which translates visual images into sounds. In addition, participants' auditory abilities and their phenomenologies were measured. Our study revealed that, after training, when asked to identify sounds, processes shared with vision were involved, as participants’ performance in sound identification was influenced by the simultaneously presented visual distractors. In addition, participants’ performance during training and their associated phenomenology depended on their auditory abilities, revealing that processing finds its roots in the input sensory modality. Our results pave the way for improving the design and learning of these devices by taking into account inter-individual differences in auditory and visual perceptual strategies.

Ears on the Hand: Reaching Three-Dimensional Targets With an Audio-Motor Device

Understanding the processes underlying sensorimotor coupling with the environment is crucial for sensorimotor rehabilitation and sensory substitution. In doing so, devices which provide novel sensory feedback consequent to body movement may be optimized in order to enhance motor performance for particular tasks. The aim of the study reported here was to investigate audio-motor coupling when the auditory experience is linked to movements of the head or the hands. The participants had to localize and reach a virtual source with the dominant hand in response to sounds. An electromagnetic system recorded the position and orientation of the participants' head and hands. This system was connected to a 3D audio system that provided binaural auditory feedback on the position of the virtual listener located on the participants' body. The listener's position was computed either from the hands or from the head. For the hand condition, the virtual listener was placed on the dominant hand (the one used to reach the target) in Experiment 1 and on the non-dominant hand, which was constrained in order to have similar amplitude and degrees of freedom as that of the head, in Experiment 2. The results revealed that, in the two experiments, the participants were able to localize a source within the 3D auditory environment. Performance varied as a function of the effector's degrees of freedom and the spatial coincidence between sensor and effector. The results also allowed characterizing the kinematics of the hand and head and how they change with audio-motor coupling condition and practice.

Integration and binding in rehabilitative sensory substitution: Increasing resolution using a new Zooming-in approach

PURPOSE

To visually perceive our surroundings we constantly move our eyes and focus on particular details, and then integrate them into a combined whole. Current visual rehabilitation methods, both invasive, like bionic-eyes and non-invasive, like Sensory Substitution Devices (SSDs), down-sample visual stimuli into low-resolution images. Zooming-in to sub-parts of the scene could potentially improve detail perception. Can congenitally blind individuals integrate a 'visual' scene when offered this information via different sensory modalities, such as audition? Can they integrate visual information -perceived in parts - into larger percepts despite never having had any visual experience?

METHODS

We explored these questions using a zooming-in functionality embedded in the EyeMusic visual-to-auditory SSD. Eight blind participants were tasked with identifying cartoon faces by integrating their individual components recognized via the EyeMusic's zooming mechanism.

RESULTS

After specialized training of just 6-10 hours, blind participants successfully and actively integrated facial features into cartooned identities in 79±18% of the trials in a highly significant manner, (chance level 10% ; rank-sum P <  1.55E-04).

CONCLUSIONS

These findings show that even users who lacked any previous visual experience whatsoever can indeed integrate this visual information with increased resolution. This potentially has important practical visual rehabilitation implications for both invasive and non-invasive methods.

Sensorimotor strategies for recognizing geometrical shapes: a comparative study with different sensory substitution devices

The sensorimotor approach proposes that perception is constituted by the mastery of lawful sensorimotor regularities or sensorimotor contingencies (SMCs), which depend on specific bodily characteristics and on actions possibilities that the environment enables and constrains. Sensory substitution devices (SSDs) provide the user information about the world typically corresponding to one sensory modality through the stimulation of another modality. We investigate how perception emerges in novice adult participants equipped with vision-to-auditory SSDs while solving a simple geometrical shape recognition task. In particular, we examine the distinction between apparatus-related SMCs (those originating mostly in properties of the perceptual system) and object-related SMCs (those mostly connected with the perceptual task). We study the sensorimotor strategies employed by participants in three experiments with three different SSDs: a minimalist head-mounted SSD, a traditional, also head-mounted SSD (the vOICe) and an enhanced, hand-held echolocation device. Motor activity and fist-person data are registered and analyzed. Results show that participants are able to quickly learn the necessary skills to distinguish geometric shapes. Comparing the sensorimotor strategies utilized with each SSD we identify differential features of the sensorimotor patterns attributable mostly to the device, which account for the emergence of apparatus-based SMCs. These relate to differences in sweeping strategies between SSDs. We identify, also, components related to the emergence of object-related SMCs. These relate mostly to exploratory movements around the border of a shape. The study provides empirical support for SMC theory and discusses considerations about the nature of perception in sensory substitution.

Reading the World through the Skin and Ears: A New Perspective on Sensory Substitution

Sensory substitution devices aim at replacing or assisting one or several functions of a deficient sensory modality by means of another sensory modality. Despite the numerous studies and research programs devoted to their development and integration, sensory substitution devices have failed to live up to their goal of allowing one to "see with the skin" (White et al., 1970) or to "see with the brain" (Bach-y-Rita et al., 2003). These somewhat peremptory claims, as well as the research conducted so far, are based on an implicit perceptual paradigm. Such perceptual assumption accepts the equivalence between using a sensory substitution device and perceiving through a particular sensory modality. Our aim is to provide an alternative model, which defines sensory substitution as being closer to culturally implemented cognitive extensions of existing perceptual skills such as reading. In this article, we will show why the analogy with reading provides a better explanation of the actual findings, that is, both of the positive results achieved and of the limitations noticed across the field of research on sensory substitution. The parallel with the most recent two-route and interactive models of reading (e.g., Dehaene et al., 2005) generates a radically new way of approaching these results, by stressing the dependence of integration on the existing perceptual-semantic route. In addition, the present perspective enables us to generate innovative research questions and specific predictions which set the stage for future work.

The Enactive Torch: A New Tool for the Science of Perception

The cognitive sciences are increasingly coming to terms with the embodied, embedded, extended, and experiential aspects of the mind. Exemplifying this shift, the enactive approach points to an essential role of goal-directed bodily activity in the generation of meaningful perceptual experience, i.e., sense-making. Here, building on recent insights into the transformative effects of practical tool-use, we make use of the enactive approach in order to provide a definition of an enactive interface in terms of augmented sense-making. We introduce such a custom-built interface, the Enactive Torch, and present a study of its experiential effects. The results demonstrate that the user experience is not adequately captured by any standardly assumed perceptual modality; rather, it is a new feeling that is mediated by the design of the device and shaped by the overall situation of the task. Taken together these findings show that there is much to be gained by synergies between engineering and the cognitive sciences in the creation of new experience-centered technology. We suggest that the guiding principle should be the design of interfaces that serve as a transparent medium for augmenting our natural skills of interaction with the world, instead of requiring conscious attention to the interface as an opaque object in the world.

The science of interpersonal touch: an overview

Surprisingly little scientific research has been conducted on the topic of interpersonal touch over the years, despite the importance of touch in our everyday social interactions from birth through to adulthood and old age. In this review, we critically evaluate the results of the research on this topic that have emerged from disciplines, such as cognitive and social psychology, neuroscience, and cultural anthropology. We highlight some of the most important advances to have been made in our understanding of this topic: For example, research has shown that interpersonal tactile stimulation provides an effective means of influencing people's social behaviors (such as modulating their tendency to comply with requests, in affecting people's attitudes toward specific services, in creating bonds between couples or groups, and in strengthening romantic relationships), regardless of whether or not the tactile contact itself can be remembered explicitly. What is more, interpersonal touch can be used to communicate emotion in a manner similar to that demonstrated previously in vision and audition. The recent growth of studies investigating the potential introduction of tactile sensations to long-distance communication technologies (by means of mediated or 'virtual' touch) are also reviewed briefly. Finally, we highlight the synergistic effort that will be needed by researchers in different disciplines if we are to develop a more complete understanding of interpersonal touch in the years to come.