Evaluation of an Audio-haptic Sensory Substitution Device for Enhancing Spatial Awareness for the Visually Impaired

Towards sensory substitution and augmentation: Mapping visual distance to audio and tactile frequency

Multimodal perception is the predominant means by which individuals experience and interact with the world. However, sensory dysfunction or loss can significantly impede this process. In such cases, cross-modality research offers valuable insight into how we can compensate for these sensory deficits through sensory substitution. Although sight and hearing are both used to estimate the distance to an object (e.g., by visual size and sound volume) and the perception of distance is an important element in navigation and guidance, it is not widely studied in cross-modal research. We investigate the relationship between audio and vibrotactile frequencies (in the ranges 47-2,764 Hz and 10-99 Hz, respectively) and distances uniformly distributed in the range 1-12 m. In our experiments participants mapped the distance (represented by an image of a model at that distance) to a frequency via adjusting a virtual tuning knob. The results revealed that the majority (more than 76%) of participants demonstrated a strong negative monotonic relationship between frequency and distance, across both vibrotactile (represented by a natural log function) and auditory domains (represented by an exponential function). However, a subgroup of participants showed the opposite positive linear relationship between frequency and distance. The strong cross-modal sensory correlation could contribute to the development of assistive robotic technologies and devices to augment human perception. This work provides the fundamental foundation for future assisted HRI applications where a mapping between distance and frequency is needed, for example for people with vision or hearing loss, drivers with loss of focus or response delay, doctors undertaking teleoperation surgery, and users in augmented reality (AR) or virtual reality (VR) environments.

A direct comparison of sound and vibration as sources of stimulation for a sensory substitution glove

Sensory substitution devices (SSDs) facilitate the detection of environmental information through enhancement of touch and/or hearing capabilities. Research has demonstrated that several tasks can be successfully completed using acoustic, vibrotactile, and multimodal devices. The suitability of a substituting modality is also mediated by the type of information required to perform the specific task. The present study tested the adequacy of touch and hearing in a grasping task by utilizing a sensory substitution glove. The substituting modalities inform, through increases in stimulation intensity, about the distance between the fingers and the objects. A psychophysical experiment of magnitude estimation was conducted. Forty blindfolded sighted participants discriminated equivalently the intensity of both vibrotactile and acoustic stimulation, although they experienced some difficulty with the more intense stimuli. Additionally, a grasping task involving cylindrical objects of varying diameters, distances and orientations was performed. Thirty blindfolded sighted participants were divided into vibration, sound, or multimodal groups. High performance was achieved (84% correct grasps) with equivalent success rate between groups. Movement variables showed more precision and confidence in the multimodal condition. Through a questionnaire, the multimodal group indicated their preference for using a multimodal SSD in daily life and identified vibration as their primary source of stimulation. These results demonstrate that there is an improvement in performance with specific-purpose SSDs, when the necessary information for a task is identified and coupled with the delivered stimulation. Furthermore, the results suggest that it is possible to achieve functional equivalence between substituting modalities when these previous steps are met.

New frontiers of retinal therapeutic intervention: a critical analysis of novel approaches

A recent wave of pharmacologic and technologic innovations has revolutionized our management of retinal diseases. Many of these advancements have demonstrated efficacy and can increase the quality of life while potentially reducing complications and decreasing the burden of care for patients. Some advances, such as longer-acting anti-vascular endothelial growth factor agents, port delivery systems, gene therapy, and retinal prosthetics have been approved by the US Food and Drug Administration, and are available for clinical use. Countless other therapeutics are in various stages of development, promising a bright future for further improvements in the management of the retinal disease. Herein, we have highlighted several important novel therapies and therapeutic approaches and examine the opportunities and limitations offered by these innovations at the new frontier. KEY MESSAGESNumerous pharmacologic and technologic advancements have been emerging, providing a higher treatment efficacy while decreasing the burden and associated side effects.Anti-vascular endothelial growth factor (anti-VEGF) and its longer-acting agents have dramatically improved visual outcomes and have become a mainstay treatment in various retinal diseases.Gene therapy and retinal prosthesis implantation in the treatment of congenital retinal dystrophy can accomplish the partial restoration of vision and improved daily function in patients with blindness, an unprecedented success in the field of retina.

The MAPS: Toward a Novel Mobility Assistance System for Visually Impaired People

This paper introduces the design of a novel indoor and outdoor mobility assistance system for visually impaired people. This system is named the MAPS (Mobility Assistance Path Planning and orientation in Space), and it is based on the theoretical frameworks of mobility and spatial cognition. Its originality comes from the assistance of two main functions of navigation: locomotion and wayfinding. Locomotion involves the ability to avoid obstacles, while wayfinding involves the orientation in space and ad hoc path planning in an (unknown) environment. The MAPS architecture proposes a new low-cost system for indoor–outdoor cognitive mobility assistance, relying on two cooperating hardware feedbacks: the Force Feedback Tablet (F2T) and the TactiBelt. F2T is an electromechanical tablet using haptic effects that allow the exploration of images and maps. It is used to assist with maps’ learning, space awareness emergence, path planning, wayfinding and effective journey completion. It helps a VIP construct a mental map of their environment. TactiBelt is a vibrotactile belt providing active support for the path integration strategy while navigating; it assists the VIP localize the nearest obstacles in real-time and provides the ego-directions to reach the destination. Technology used for acquiring the information about the surrounding space is based on vision (cameras) and is defined with the localization on a map. The preliminary evaluations of the MAPS focused on the interaction with the environment and on feedback from the users (blindfolded participants) to confirm its effectiveness in a simulated environment (a labyrinth). Those lead-users easily interpreted the system’s provided data that they considered relevant for effective independent navigation.

Emerging technologies and their potential for generating new assistive technologies

Limited access to assistive technology (AT) is a well-recognized global challenge. Emerging technologies have potential to develop new assistive products and bridge some of the gaps in access to AT. However, limited analyses exist on the potential of these technologies in the AT field. This paper describes a study that aimed to provide an overview of emerging technological developments and their potential for the AT field. It involved conducting a gray literature review and patent analysis to create an overview of the emerging enabling technologies that may foster the development of new AT products and services and identify emerging AT applications. The analysis identified seven enabling technologies that are relevant to the AT field. These are artificial intelligence, emerging human-computer interfaces, sensor technology, robotics, advances in connectivity and computing, additive manufacturing and new materials. Whilst there are over 3.7 million patents related to these enabling technologies, only a fraction of them - 11,000 patents were identified in the analysis specifically related to AT (0.3%). The paper presents some of the promising examples. Overall, the results indicate that there is an enormous potential for new AT solutions that capitalize on emerging technological advances.

Blindness and the Reliability of Downwards Sensors to Avoid Obstacles: A Study with the EyeCane

Vision loss has dramatic repercussions on the quality of life of affected people, particularly with respect to their orientation and mobility. Many devices are available to help blind people to navigate in their environment. The EyeCane is a recently developed electronic travel aid (ETA) that is inexpensive and easy to use, allowing for the detection of obstacles lying ahead within a 2 m range. The goal of this study was to investigate the potential of the EyeCane as a primary aid for spatial navigation. Three groups of participants were recruited: early blind, late blind, and sighted. They were first trained with the EyeCane and then tested in a life-size obstacle course with four obstacles types: cube, door, post, and step. Subjects were requested to cross the corridor while detecting, identifying, and avoiding the obstacles. Each participant had to perform 12 runs with 12 different obstacles configurations. All participants were able to learn quickly to use the EyeCane and successfully complete all trials. Amongst the various obstacles, the step appeared to prove the hardest to detect and resulted in more collisions. Although the EyeCane was effective for detecting obstacles lying ahead, its downward sensor did not reliably detect those on the ground, rendering downward obstacles more hazardous for navigation.

Brain-Machine Interfaces to Assist the Blind

The loss or absence of vision is probably one of the most incapacitating events that can befall a human being. The importance of vision for humans is also reflected in brain anatomy as approximately one third of the human brain is devoted to vision. It is therefore unsurprising that throughout history many attempts have been undertaken to develop devices aiming at substituting for a missing visual capacity. In this review, we present two concepts that have been prevalent over the last two decades. The first concept is sensory substitution, which refers to the use of another sensory modality to perform a task that is normally primarily sub-served by the lost sense. The second concept is cross-modal plasticity, which occurs when loss of input in one sensory modality leads to reorganization in brain representation of other sensory modalities. Both phenomena are training-dependent. We also briefly describe the history of blindness from ancient times to modernity, and then proceed to address the means that have been used to help blind individuals, with an emphasis on modern technologies, invasive (various type of surgical implants) and non-invasive devices. With the advent of brain imaging, it has become possible to peer into the neural substrates of sensory substitution and highlight the magnitude of the plastic processes that lead to a rewired brain. Finally, we will address the important question of the value and practicality of the available technologies and future directions.

Navigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study

PURPOSE

In this study, we investigate to what degree augmented reality technology can be used to create and evaluate a visual-to-auditory sensory substitution device to improve the performance of blind persons in navigation and recognition tasks.

METHODS

A sensory substitution algorithm that translates 3D visual information into audio feedback was designed. This algorithm was integrated in an augmented reality based mobile phone application. Using the mobile device as sensory substitution device, a study with blind participants (n = 7) was performed. The participants navigated through pseudo-randomized obstacle courses using either the sensory substitution device, a white cane or a combination of both. In a second task, virtual 3D objects and structures had to be identified by the participants using the same sensory substitution device.

RESULTS

The realized application for mobile devices enabled participants to complete the navigation and object recognition tasks in an experimental environment already within the first trials without previous training. This demonstrates the general feasibility and low entry barrier of the designed sensory substitution algorithm. In direct comparison to the white cane, within the study duration of ten hours the sensory substitution device did not offer a statistically significant improvement in navigation.

Multisensory inclusive design with sensory substitution

Sensory substitution techniques are perceptual and cognitive phenomena used to represent one sensory form with an alternative. Current applications of sensory substitution techniques are typically focused on the development of assistive technologies whereby visually impaired users can acquire visual information via auditory and tactile cross-modal feedback. But despite their evident success in scientific research and furthering theory development in cognition, sensory substitution techniques have not yet gained widespread adoption within sensory-impaired populations. Here we argue that shifting the focus from assistive to mainstream applications may resolve some of the current issues regarding the use of sensory substitution devices to improve outcomes for those with disabilities. This article provides a tutorial guide on how to use research into multisensory processing and sensory substitution techniques from the cognitive sciences to design new inclusive cross-modal displays. A greater focus on developing inclusive mainstream applications could lead to innovative technologies that could be enjoyed by every person.

Sensory augmentation to aid training with retinal prostheses

OBJECTIVE

Retinal prosthesis recipients require rehabilitative training to learn the non-intuitive nature of prosthetic 'phosphene vision'. This study investigated whether the addition of auditory cues, using The vOICe sensory substitution device (SSD), could improve functional performance with simulated phosphene vision.

APPROACH

Forty normally sighted subjects completed two visual tasks under three conditions. The phosphene condition converted the image to simulated phosphenes displayed on a virtual reality headset. The SSD condition provided auditory information via stereo headphones, translating the image into sound. Horizontal information was encoded as stereo timing differences between ears, vertical information as pitch, and pixel intensity as audio intensity. The third condition combined phosphenes and SSD. Tasks comprised light localisation from the Basic Assessment of Light and Motion (BaLM) and the Tumbling-E from the Freiburg Acuity and Contrast Test (FrACT). To examine learning effects, twenty of the forty subjects received SSD training prior to assessment.

MAIN RESULTS

Combining phosphenes with auditory SSD provided better light localisation accuracy than either phosphenes or SSD alone, suggesting a compound benefit of integrating modalities. Although response times for SSD-only were significantly longer than all other conditions, combined condition response times were as fast as phosphene-only, highlighting that audio-visual integration provided both response time and accuracy benefits. Prior SSD training provided a benefit to localisation accuracy and speed in SSD-only (as expected) and Combined conditions compared to untrained SSD-only. Integration of the two modalities did not improve spatial resolution task performance, with resolution limited to that of the higher resolution modality (SSD).

SIGNIFICANCE

Combining phosphene (visual) and SSD (auditory) modalities was effective even without SSD training and led to an improvement in light localisation accuracy and response times. Spatial resolution performance was dominated by auditory SSD. The results suggest there may be a benefit to including auditory cues when training vision prosthesis recipients.

Tactile Working Memory Capacity of Users Who Are Blind in an Electronic Travel Aid Application with a Vibration Belt

Electronic travel aids (ETAs) can increase the safety and comfort of pedestrians who have a visual impairment by displaying obstacles through a vibrotactile navigation belt. Building a complete picture of relevant obstacles and finding a safe route requires ETA users to integrate vibrotactile cues over time and space in their tactile working memory. Previous research suggests that the sense of touch exhibits a working memory that has characteristics similar to vision and audition. However, the capacity of the tactile working memory and the effects of secondary tasks are still under-researched. We investigated tactile working memory capacity of 14 adolescent participants who are blind in an immediate, whole report recall test. Participants received trials consisting of one to five vibration patterns presented sequentially at different locations on their torso representing obstacles with a direction (vibration location) and distance (vibration pattern). Recall performance was assessed under four conditions: baseline and with distracting background sounds and/or while walking with the long cane. Both walking and ignoring distracting sounds are relevant for everyday use of an ETA and were expected to decrease memory performance. We calculated the 75% correct scores for two memory performance measures: the number of items in a trial (numerosity), and item location and pattern correct. In the baseline condition, the scores were close to ceiling (i.e., 5 items). However, in the presence of distracting sounds and while walking, the scores were reduced to 3.2 items for numerosity and 1.6 items for location and identity correct. We recommend using 2 items as the maximum tactile working memory load in an applied setting unless users are trained and/or can adopt their strategy without unacceptable costs, such as reducing their walking speed.

Audomni: Super-Scale Sensory Supplementation to Increase the Mobility of Blind and Low-Vision Individuals-A Pilot Study

OBJECTIVE

Blindness and low vision have severe effects on individuals' quality of life and socioeconomic cost; a main contributor of which is a prevalent and acutely decreased mobility level. To alleviate this, numerous technological solutions have been proposed in the last 70 years; however, none has become widespread.

METHOD

In this paper, we introduce the vision-to-audio, super-scale sensory substitution/supplementation device Audomni; we address the field-encompassing issues of ill-motivated and overabundant test methodologies and metrics; and we utilize our proposed Desire of Use model to evaluate proposed pilot user tests, their results, and Audomni itself.

RESULTS

Audomni holds a spatial resolution of 80 x 60 pixels at ~1.2° angular resolution and close to real-time temporal resolution, outdoor-viable technology, and several novel differentiation methods. The tests indicated that Audomni has a low learning curve, and several key mobility subtasks were accomplished; however, the tests would benefit from higher real-life motivation and data collection affordability.

CONCLUSION

Audomni shows promise to be a viable mobility device - with some addressable issues. Employing Desire of Use to design future tests should provide both high real-life motivation and relevance to them.

SIGNIFICANCE

As far as we know, Audomni features the greatest information conveyance rate in the field, yet seems to offer comprehensible and fairly intuitive sonification; this work is also the first to utilize Desire of Use as a tool to evaluate user tests, a device, and to lay out an overarching project aim.

Measuring relative vibrotactile spatial acuity: effects of tactor type, anchor points and tactile anisotropy

Vibrotactile displays can compensate for the loss of sensory function of people with permanent or temporary deficiencies in vision, hearing, or balance, and can augment the immersive experience in virtual environments for entertainment, or professional training. This wide range of potential applications highlights the need for research on the basic psychophysics of mechanisms underlying human vibrotactile perception. One key consideration when designing tactile displays is determining the minimal possible spacing between tactile motors (tactors), by empirically assessing the maximal throughput of the skin, or, in other words, vibrotactile spatial acuity. Notably, such estimates may vary by tactor type. We assessed vibrotactile spatial acuity in the lower thoracic region for three different tactor types, each mounted in a 4 × 4 array with center-to-center inter-tactor distances of 25 mm, 20 mm, and 10 mm. Seventeen participants performed a relative three-alternative forced-choice point localization task with successive tactor activation for both vertical and horizontal stimulus presentation. The results demonstrate that specific tactor characteristics (frequency, acceleration, contact area) significantly affect spatial acuity measurements, highlighting that the results of spatial acuity measurements may only apply to the specific tactors tested. Furthermore, our results reveal an anisotropy in vibrotactile perception, with higher spatial acuity for horizontal than for vertical stimulus presentation. The findings allow better understanding of vibrotactile spatial acuity and can be used for formulating guidelines for the design of tactile displays, such as regarding inter-tactor spacing, choice of tactor type, and direction of stimulus presentation.