Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception

Comparing episodic memory outcomes from walking augmented reality and stationary virtual reality encoding experiences

Episodic Memory (EM) is the neurocognitive capacity to consciously recollect personally experienced events in specific spatio-temporal contexts. Although the relevance of spatial and temporal information is widely acknowledged in the EM literature, it remains unclear whether and how EM performance and organisation is modulated by self-motion, and by motor- and visually- salient environmental features (EFs) of the encoding environment. This study examines whether and how EM is modulated by locomotion and the EFs encountered in a controlled lifelike learning route within a large-scale building. Twenty-eight healthy participants took part in a museum-tour encoding task implemented in walking Augmented Reality (AR) and stationary Virtual Reality (VR) conditions. EM performance and organisation were assessed immediately and 48-hours after trials using a Remember/Familiar recognition paradigm. Results showed a significant positive modulation effect of locomotion on distinctive EM aspects. Findings highlighted a significant performance enhancement effect of stairway-adjacent locations compared to dead-end and mid-route stimuli-presentation locations. The results of this study may serve as design criteria to facilitate neurocognitive rehabilitative interventions of EM. The underlying technological framework developed for this study represents a novel and ecologically sound method for evaluating EM processes in lifelike situations, allowing researchers a naturalistic perspective into the complex nature of EM.

Enhancing human-human musical interaction through kinesthetic haptic feedback using wearable exoskeletons: theoretical foundations, validation scenarios, and limitations

In this perspective paper, we explore the use of haptic feedback to enhance human-human interaction during musical tasks. We start by providing an overview of the theoretical foundation that underpins our approach, which is rooted in the embodied music cognition framework, and by briefly presenting the concepts of action-perception loop, sensorimotor coupling and entrainment. Thereafter, we focus on the role of haptic information in music playing and we discuss the use of wearable technologies, namely lightweight exoskeletons, for the exchange of haptic information between humans. We present two experimental scenarios in which the effectiveness of this technology for enhancing musical interaction and learning might be validated. Finally, we briefly discuss some of the theoretical and pedagogical implications of the use of technologies for haptic communication in musical contexts, while also addressing the potential barriers to the widespread adoption of exoskeletons in such contexts.

Getting LOST: A conceptual framework for supporting and enhancing spatial navigation in aging

Spatial navigation is more difficult and effortful for older than younger individuals, a shift which occurs for a variety of neurological, physical, and cognitive reasons associated with aging. Despite a large body of evidence documenting age-related deficits in spatial navigation, comparatively less research addresses how to facilitate more effective navigation behavior for older adults. Since navigation challenges arise for a variety of reasons in old age, a one-size-fits-all solution is unlikely to work. Here, we introduce a framework for the variety of spatial navigation challenges faced in aging, which we call LOST-Location, Orientation, Spatial mapping, and Transit. The LOST framework builds on evidence from the cognitive neuroscience of spatial navigation, which reveals distinct components underpinning human wayfinding. We evaluate research on navigational aids-devices and depictions-which help people find their way around; and we reflect on how navigation aids solve (or fail to solve) specific wayfinding difficulties faced by older adults. In summary, we emphasize a bespoke approach to improving spatial navigation in aging, which focuses on tailoring navigation solutions to specific navigation challenges. Our hope is that by providing precise support to older navigators, navigation opportunities can facilitate independence and exploration, while minimizing the danger of becoming lost. We conclude by delineating critical knowledge gaps in how to improve older adults' spatial navigation capacities that the novel LOST framework could guide to address. This article is categorized under: Psychology > Development and Aging Neuroscience > Cognition Neuroscience > Behavior.

Sensory augmentation for a rapid motor task in a multisensory environment

BACKGROUND

Sensory substitution and augmentation systems (SSASy) seek to either replace or enhance existing sensory skills by providing a new route to access information about the world. Tests of such systems have largely been limited to untimed, unisensory tasks.

OBJECTIVE

To test the use of a SSASy for rapid, ballistic motor actions in a multisensory environment.

METHODS

Participants played a stripped-down version of air hockey in virtual reality with motion controls (Oculus Touch). They were trained to use a simple SASSy (novel audio cue) for the puck's location. They were tested on ability to strike an oncoming puck with the SASSy, degraded vision, or both.

RESULTS

Participants coordinated vision and the SSASy to strike the target with their hand more consistently than with the best single cue alone, t(13) = 9.16, p <.001, Cohen's d = 2.448.

CONCLUSIONS

People can adapt flexibly to using a SSASy in tasks that require tightly timed, precise, and rapid body movements. SSASys can augment and coordinate with existing sensorimotor skills rather than being limited to replacement use cases - in particular, there is potential scope for treating moderate vision loss. These findings point to the potential for augmenting human abilities, not only for static perceptual judgments, but in rapid and demanding perceptual-motor tasks.

Improved Spatial Knowledge Acquisition through Sensory Augmentation

Sensory augmentation provides novel opportunities to broaden our knowledge of human perception through external sensors that record and transmit information beyond natural perception. To assess whether such augmented senses affect the acquisition of spatial knowledge during navigation, we trained a group of 27 participants for six weeks with an augmented sense for cardinal directions called the feelSpace belt. Then, we recruited a control group that did not receive the augmented sense and the corresponding training. All 53 participants first explored the Westbrook virtual reality environment for two and a half hours spread over five sessions before assessing their spatial knowledge in four immersive virtual reality tasks measuring cardinal, route, and survey knowledge. We found that the belt group acquired significantly more accurate cardinal and survey knowledge, which was measured in pointing accuracy, distance, and rotation estimates. Interestingly, the augmented sense also positively affected route knowledge, although to a lesser degree. Finally, the belt group reported a significant increase in the use of spatial strategies after training, while the groups' ratings were comparable at baseline. The results suggest that six weeks of training with the feelSpace belt led to improved survey and route knowledge acquisition. Moreover, the findings of our study could inform the development of assistive technologies for individuals with visual or navigational impairments, which may lead to enhanced navigation skills and quality of life.

Extended training improves the accuracy and efficiency of goal-directed reaching guided by supplemental kinesthetic vibrotactile feedback

Prior studies have shown that the accuracy and efficiency of reaching can be improved using novel sensory interfaces to apply task-specific vibrotactile feedback (VTF) during movement. However, those studies have typically evaluated performance after less than 1 h of training using VTF. Here, we tested the effects of extended training using a specific form of vibrotactile cues-supplemental kinesthetic VTF-on the accuracy and temporal efficiency of goal-directed reaching. Healthy young adults performed planar reaching with VTF encoding of the moving hand's instantaneous position, applied to the non-moving arm. We compared target capture errors and movement times before, during, and after approximately 10 h (20 sessions) of training on the VTF-guided reaching task. Initial performance of VTF-guided reaching showed that people were able to use supplemental VTF to improve reaching accuracy. Performance improvements were retained from one training session to the next. After 20 sessions of training, the accuracy and temporal efficiency of VTF-guided reaching were equivalent to or better than reaches performed with only proprioception. However, hand paths during VTF-guided reaching exhibited a persistent strategy where movements were decomposed into discrete sub-movements along the cardinal axes of the VTF display. We also used a dual-task condition to assess the extent to which performance gains in VTF-guided reaching resist dual-task interference. Dual-tasking capability improved over the 20 sessions, such that the primary VTF-guided reaching and a secondary choice reaction time task were performed with increasing concurrency. Thus, VTF-guided reaching is a learnable skill in young adults, who can achieve levels of accuracy and temporal efficiency equaling or exceeding those observed during movements guided only by proprioception. Future studies are warranted to explore learnability in older adults and patients with proprioceptive deficits, who might benefit from using wearable sensory augmentation technologies to enhance control of arm movements.

Bridging a sensory gap between deaf and hearing people–A plea for a situated design approach to sensory augmentation

Deaf and hearing people can encounter challenges when communicating with one another in everyday situations. Although problems in verbal communication are often seen as the main cause, such challenges may also result from sensory differences between deaf and hearing people and their impact on individual understandings of the world. That is, challenges arising from a sensory gap. Proposals for innovative communication technologies to address this have been met with criticism by the deaf community. They are mostly designed to enhance deaf people's understanding of the verbal cues that hearing people rely on, but omit many critical sensory signals that deaf people rely on to understand (others in) their environment and to which hearing people are not tuned to. In this perspective paper, sensory augmentation, i.e., technologically extending people's sensory capabilities, is put forward as a way to bridge this sensory gap: (1) by tuning to the signals deaf people rely on more strongly but are commonly missed by hearing people, and vice versa, and (2) by sensory augmentations that enable deaf and hearing people to sense signals that neither person is able to normally sense. Usability and user-acceptance challenges, however, lie ahead of realizing the alleged potential of sensory augmentation for bridging the sensory gap between deaf and hearing people. Addressing these requires a novel approach to how such technologies are designed. We contend this requires a situated design approach.

Retention Effects of Long-Term Balance Training with Vibrotactile Sensory Augmentation in Healthy Older Adults

Vibrotactile sensory augmentation (SA) decreases postural sway during real-time use; however, limited studies have investigated the long-term effects of training with SA. This study assessed the retention effects of long-term balance training with and without vibrotactile SA among community-dwelling healthy older adults, and explored brain-related changes due to training with SA. Sixteen participants were randomly assigned to the experimental group (EG) or control group (CG), and trained in their homes for eight weeks using smart-phone balance trainers. The EG received vibrotactile SA. Balance performance was assessed before, and one week, one month, and six months after training. Functional MRI (fMRI) was recorded before and one week after training for four participants who received vestibular stimulation. Both groups demonstrated significant improvement of SOT composite and MiniBESTest scores, and increased vestibular reliance. Only the EG maintained a minimal detectable change of 8 points in SOT scores six months post-training and greater improvements than the CG in MiniBESTest scores one month post-training. The fMRI results revealed a shift from activation in the vestibular cortex pre-training to increased activity in the brainstem and cerebellum post-training. These findings showed that additional balance improvements were maintained for up to six months post-training with vibrotactile SA for community-dwelling healthy older adults.

Multisensory Proximity and Transition Cues for Improving Target Awareness in Narrow Field of View Augmented Reality Displays

Augmented reality applications allow users to enrich their real surroundings with additional digital content. However, due to the limited field of view of augmented reality devices, it can sometimes be difficult to become aware of newly emerging information inside or outside the field of view. Typical visual conflicts like clutter and occlusion of augmentations occur and can be further aggravated especially in the context of dense information spaces. In this article, we evaluate how multisensory cue combinations can improve the awareness for moving out-of-view objects in narrow field of view augmented reality displays. We distinguish between proximity and transition cues in either visual, auditory or tactile manner. Proximity cues are intended to enhance spatial awareness of approaching out-of-view objects while transition cues inform the user that the object just entered the field of view. In study 1, user preference was determined for 6 different cue combinations via forced-choice decisions. In study 2, the 3 most preferred modes were then evaluated with respect to performance and awareness measures in a divided attention reaction task. Both studies were conducted under varying noise levels. We show that on average the Visual-Tactile combination leads to 63% and Audio-Tactile to 65% faster reactions to incoming out-of-view augmentations than their Visual-Audio counterpart, indicating a high usefulness of tactile transition cues. We further show a detrimental effect of visual and audio noise on performance when feedback included visual proximity cues. Based on these results, we make recommendations to determine which cue combination is appropriate for which application.

Impact of a Vibrotactile Belt on Emotionally Challenging Everyday Situations of the Blind

Spatial orientation and navigation depend primarily on vision. Blind people lack this critical source of information. To facilitate wayfinding and to increase the feeling of safety for these people, the "feelSpace belt" was developed. The belt signals magnetic north as a fixed reference frame via vibrotactile stimulation. This study investigates the effect of the belt on typical orientation and navigation tasks and evaluates the emotional impact. Eleven blind subjects wore the belt daily for seven weeks. Before, during and after the study period, they filled in questionnaires to document their experiences. A small sub-group of the subjects took part in behavioural experiments before and after four weeks of training, i.e., a straight-line walking task to evaluate the belt's effect on keeping a straight heading, an angular rotation task to examine effects on egocentric orientation, and a triangle completion navigation task to test the ability to take shortcuts. The belt reduced subjective discomfort and increased confidence during navigation. Additionally, the participants felt safer wearing the belt in various outdoor situations. Furthermore, the behavioural tasks point towards an intuitive comprehension of the belt. Altogether, the blind participants benefited from the vibrotactile belt as an assistive technology in challenging everyday situations.

Backward spatial perception can be augmented through a novel visual-to-auditory sensory substitution algorithm

Can humans extend and augment their natural perceptions during adulthood? Here, we address this fascinating question by investigating the extent to which it is possible to successfully augment visual spatial perception to include the backward spatial field (a region where humans are naturally blind) via other sensory modalities (i.e., audition). We thus developed a sensory-substitution algorithm, the “Topo-Speech” which conveys identity of objects through language, and their exact locations via vocal-sound manipulations, namely two key features of visual spatial perception. Using two different groups of blindfolded sighted participants, we tested the efficacy of this algorithm to successfully convey location of objects in the forward or backward spatial fields following ~ 10 min of training. Results showed that blindfolded sighted adults successfully used the Topo-Speech to locate objects on a 3 × 3 grid either positioned in front of them (forward condition), or behind their back (backward condition). Crucially, performances in the two conditions were entirely comparable. This suggests that novel spatial sensory information conveyed via our existing sensory systems can be successfully encoded to extend/augment human perceptions. The implications of these results are discussed in relation to spatial perception, sensory augmentation and sensory rehabilitation.

Topographic maps and neural tuning for sensory substitution dimensions learned in adulthood in a congenital blind subject

Topographic maps, a key principle of brain organization, emerge during development. It remains unclear, however, whether topographic maps can represent a new sensory experience learned in adulthood. MaMe, a congenitally blind individual, has been extensively trained in adulthood for perception of a 2D auditory-space (soundscape) where the y- and x-axes are represented by pitch and time, respectively. Using population receptive field mapping we found neural populations tuned topographically to pitch, not only in the auditory cortices but also in the parietal and occipito-temporal cortices. Topographic neural tuning to time was revealed in the parietal and occipito-temporal cortices. Some of these maps were found to represent both axes concurrently, enabling MaMe to represent unique locations in the soundscape space. This case study provides proof of concept for the existence of topographic maps tuned to the newly learned soundscape dimensions. These results suggest that topographic maps can be adapted or recycled in adulthood to represent novel sensory experiences.

Learning of Spatial Properties of a Large-Scale Virtual City With an Interactive Map

To become acquainted with large-scale environments such as cities people combine direct experience and indirect sources such as maps. To ascertain which type of spatial knowledge is acquired by which source is difficult to evaluate. Using virtual reality enables the possibility to investigate whether knowledge is learned by direct experience or the use of a map differentially. Therefore, we designed a large virtual city, comprised of over 200 houses, and evaluated spatial knowledge acquisition after city exploration with an interactive map following one and three 30-min exploration sessions. We tested subjects' knowledge of the orientation of houses facing directions toward cardinal north, of orientations of houses facing directions relative to each other and pointing from one house to another. Our results revealed that increased familiarity after extended exploration with the map improved task accuracy. Further, it revealed task differences, caused mainly by a better accuracy in the relative orientation task than the pointing task. Time for cognitive reasoning improved overall task accuracy. Learning with our VR city map revealed an absence of distance effect, an alignment effect of tested house orientation toward map north and an angular difference effect between tested stimuli. Self-reported knowledge of cardinal directions learned in the real environment was positively correlated with task accuracy testing houses orientations toward cardinal north. Overall, our results suggest that participants learned spatial information that is directly available in the interactive map, while a spatial task that needed integration of learned knowledge stayed at lower accuracy levels.

Movement-Contingent Time Flow in Virtual Reality Causes Temporal Recalibration

Virtual reality (VR) provides a valuable research tool for studying what occurs when sensorimotor feedback loops are manipulated. Here we measured whether exposure to a novel temporal relationship between action and sensory reaction in VR causes recalibration of time perception. We asked 31 participants to perform time perception tasks where the interval of a moving probe was reproduced using continuous or discrete motor methods. These time perception tasks were completed pre- and post-exposure to dynamic VR content in a block-counterbalanced order. One group of participants experienced a standard VR task ("normal-time"), while another group had their real-world movements coupled to the flow of time in the virtual space ("movement contingent time-flow; MCTF"). We expected this novel action-perception relationship to affect continuous motor time perception performance, but not discrete motor time perception. The results indicated duration-dependent recalibration specific to a motor task involving continuous movement such that the probe intervals were under-estimated by approximately 15% following exposure to VR with the MCTF manipulation. Control tasks in VR and non-VR settings produced similar results to those of the normal-time VR group, confirming the specificity of the MCTF manipulation. The findings provide valuable insights into the potential impact of VR on sensorimotor recalibration. Understanding this process will be valuable for the development and implementation of rehabilitation practices.

Neuromorphic Vibrotactile Stimulation of Fingertips for Encoding Object Stiffness in Telepresence Sensory Substitution and Augmentation Applications

We present a tactile telepresence system for real-time transmission of information about object stiffness to the human fingertips. Experimental tests were performed across two laboratories (Italy and Ireland). In the Italian laboratory, a mechatronic sensing platform indented different rubber samples. Information about rubber stiffness was converted into on-off events using a neuronal spiking model and sent to a vibrotactile glove in the Irish laboratory. Participants discriminated the variation of the stiffness of stimuli according to a two-alternative forced choice protocol. Stiffness discrimination was based on the variation of the temporal pattern of spikes generated during the indentation of the rubber samples. The results suggest that vibrotactile stimulation can effectively simulate surface stiffness when using neuronal spiking models to trigger vibrations in the haptic interface. Specifically, fractional variations of stiffness down to 0.67 were significantly discriminated with the developed neuromorphic haptic interface. This is a performance comparable, though slightly worse, to the threshold obtained in a benchmark experiment evaluating the same set of stimuli naturally with the own hand. Our paper presents a bioinspired method for delivering sensory feedback about object properties to human skin based on contingency-mimetic neuronal models, and can be useful for the design of high performance haptic devices.

Can Limitations of Visuospatial Attention Be Circumvented? A Review

In daily life, humans are bombarded with visual input. Yet, their attentional capacities for processing this input are severely limited. Several studies have investigated factors that influence these attentional limitations and have identified methods to circumvent them. Here, we provide a review of these findings. We first review studies that have demonstrated limitations of visuospatial attention and investigated physiological correlates of these limitations. We then review studies in multisensory research that have explored whether limitations in visuospatial attention can be circumvented by distributing information processing across several sensory modalities. Finally, we discuss research from the field of joint action that has investigated how limitations of visuospatial attention can be circumvented by distributing task demands across people and providing them with multisensory input. We conclude that limitations of visuospatial attention can be circumvented by distributing attentional processing across sensory modalities when tasks involve spatial as well as object-based attentional processing. However, if only spatial attentional processing is required, limitations of visuospatial attention cannot be circumvented by distributing attentional processing. These findings from multisensory research are applicable to visuospatial tasks that are performed jointly by two individuals. That is, in a joint visuospatial task requiring object-based as well as spatial attentional processing, joint performance is facilitated when task demands are distributed across sensory modalities. Future research could further investigate how applying findings from multisensory research to joint action research may facilitate joint performance. Generally, findings are applicable to real-world scenarios such as aviation or car-driving to circumvent limitations of visuospatial attention.

Feel the way with a vibrotactile compass: Does a navigational aid aid navigation?

Knowing where north is provides a navigator with invaluable information for learning and recalling a space, particularly in places with limited navigational cues, like complex indoor environments. Although north is effectively used by orienteers, pilots, and military personnel, very little is known about whether nonexpert populations can or will use north to create an accurate representation of an indoor space. In the current study, we taught people 2 nonoverlapping routes through a complex indoor environment, with which they were not familiar-a university hospital with few windows and several turns. Along 1 route, they wore a vibrotactile compass on their arm, which vibrated continuously indicating the direction of north. Along the other route, they were only told where north was at the start of the route. At the beginning, the end, and back at the beginning of each route, participants pointed to well-known landmarks in the surrounding city and campus (external landmarks), and newly learned landmarks in the hospital (internal landmarks). We found improved performance with the compass only for external landmarks, driven by people's use of the availability of north to orient these judgments. No such improved orientation occurred for the internal landmarks. These findings reveal the utility of vibrotactile compasses for learning new indoor spaces. We speculate that such cues help users map new spaces onto familiar spaces or to familiar reference frames. (PsycINFO Database Record

Are allocentric spatial reference frames compatible with theories of Enactivism?

Theories of Enactivism propose an action-oriented approach to understand human cognition. So far, however, empirical evidence supporting these theories has been sparse. Here, we investigate whether spatial navigation based on allocentric reference frames that are independent of the observer's physical body can be understood within an action-oriented approach. Therefore, we performed three experiments testing the knowledge of the absolute orientation of houses and streets towards north, the relative orientation of two houses and two streets, respectively, and the location of houses towards each other in a pointing task. Our results demonstrate that under time pressure, the relative orientation of two houses can be retrieved more accurately than the absolute orientation of single houses. With infinite time for cognitive reasoning, the performance of the task using house stimuli increased greatly for the absolute orientation and surpassed the slightly improved performance in the relative orientation task. In contrast, with streets as stimuli participants performed under time pressure better in the absolute orientation task. Overall, pointing from one house to another house yielded the best performance. This suggests, first, that orientation and location information about houses are primarily coded in house-to-house relations, whereas cardinal information is deduced via cognitive reasoning. Second, orientation information for streets is preferentially coded in absolute orientations. Thus, our results suggest that spatial information about house and street orientation is coded differently and that house orientation and location is primarily learned in an action-oriented way, which is in line with an enactive framework for human cognition.

Is Attentional Resource Allocation Across Sensory Modalities Task-Dependent?

Human information processing is limited by attentional resources. That is, via attentional mechanisms, humans select a limited amount of sensory input to process while other sensory input is neglected. In multisensory research, a matter of ongoing debate is whether there are distinct pools of attentional resources for each sensory modality or whether attentional resources are shared across sensory modalities. Recent studies have suggested that attentional resource allocation across sensory modalities is in part task-dependent. That is, the recruitment of attentional resources across the sensory modalities depends on whether processing involves object-based attention (e.g., the discrimination of stimulus attributes) or spatial attention (e.g., the localization of stimuli). In the present paper, we review findings in multisensory research related to this view. For the visual and auditory sensory modalities, findings suggest that distinct resources are recruited when humans perform object-based attention tasks, whereas for the visual and tactile sensory modalities, partially shared resources are recruited. If object-based attention tasks are time-critical, shared resources are recruited across the sensory modalities. When humans perform an object-based attention task in combination with a spatial attention task, partly shared resources are recruited across the sensory modalities as well. Conversely, for spatial attention tasks, attentional processing does consistently involve shared attentional resources for the sensory modalities. Generally, findings suggest that the attentional system flexibly allocates attentional resources depending on task demands. We propose that such flexibility reflects a large-scale optimization strategy that minimizes the brain’s costly resource expenditures and simultaneously maximizes capability to process currently relevant information.

Sensory augmentation: integration of an auditory compass signal into human perception of space

Bio-mimetic approaches to restoring sensory function show great promise in that they rapidly produce perceptual experience, but have the disadvantage of being invasive. In contrast, sensory substitution approaches are non-invasive, but may lead to cognitive rather than perceptual experience. Here we introduce a new non-invasive approach that leads to fast and truly perceptual experience like bio-mimetic techniques. Instead of building on existing circuits at the neural level as done in bio-mimetics, we piggy-back on sensorimotor contingencies at the stimulus level. We convey head orientation to geomagnetic North, a reliable spatial relation not normally sensed by humans, by mimicking sensorimotor contingencies of distal sounds via head-related transfer functions. We demonstrate rapid and long-lasting integration into the perception of self-rotation. Short training with amplified or reduced rotation gain in the magnetic signal can expand or compress the perceived extent of vestibular self-rotation, even with the magnetic signal absent in the test. We argue that it is the reliability of the magnetic signal that allows vestibular spatial recalibration, and the coding scheme mimicking sensorimotor contingencies of distal sounds that permits fast integration. Hence we propose that contingency-mimetic feedback has great potential for creating sensory augmentation devices that achieve fast and genuinely perceptual experiences.