Acquisition of Visual Perception in Blind Adults Using the BrainPort Artificial Vision Device

Sensory-substitution based sound perception using a spinal computer-brain interface

Sensory substitution offers a promising approach to restore lost sensory functions. Here we show that spinal cord stimulation (SCS), typically used for chronic pain management, can potentially serve as a novel auditory sensory substitution device. We recruited 13 patients undergoing SCS implantation and translated everyday sound samples into personalized SCS patterns during their trial phase. In a sound identification task-where chance-level performance was 33.3%-participants ( n = 8 ) achieved a mean accuracy of 72.8% using only SCS input. We observed a weak positive correlation between stimulation bitrate and identification accuracy. A follow-up discrimination task ( n = 5 ) confirmed that reduced bitrates significantly impaired participants' ability to distinguish between consecutive SCS patterns, indicating effective processing of additional information at higher bitrates. These findings demonstrate the feasibility of using existing SCS technology to create a novel neural interface for a sound prosthesis. Our results pave the way for future research to enhance stimulation fidelity, assess long-term training effects, and explore integration with other auditory aids for comprehensive hearing rehabilitation.

Visuo-spatial imagery in dreams of congenitally and early blind: a systematic review

Background

The presence of visual imagery in dreams of congenitally blind people has long been a matter of substantial controversy. We set to systematically review body of published work on the presence and nature of oneiric visuo-spatial impressions in congenitally and early blind subjects across different areas of research, from experimental psychology, functional neuroimaging, sensory substitution, and sleep research.

Methods

Relevant studies were identified using the following databases: EMBASE, MEDLINE and PsychINFO.

Results

Studies using diverse imaging techniques and sensory substitution devices broadly suggest that the "blind" occipital cortex may be able to integrate non-visual sensory inputs, and thus possibly also generate visuo-spatial impressions. Visual impressions have also been reported by blind subjects who had near-death or out-of-body experiences.

Conclusion

Deciphering the mechanistic nature of these visual impression could open new possibility in utilization of neuroplasticity and its potential role for treatment of neurodisability.

Optimization and Validation of a Virtual Reality Orientation and Mobility Test for Inherited Retinal Degenerations

Purpose

To optimize a virtual reality (VR) orientation and mobility (O&M) test of functional vision in patients with inherited retinal degenerations (IRDs).

Methods

We developed an O&M test using commercially available VR hardware and custom-generated software. Normally sighted subjects (n = 20, ages = 14-67 years) and patients with IRDs (n = 29, ages = 15-63 years) participated. Individuals followed a dim red arrow path to a "course exit," while trying to identify nine obstacles adjacent to, or directly in their path. Dark-adapted subjects completed 35 randomly selected VR courses at increasing luminances, twice per luminance step, binocularly, and uni-ocularly. Performance was graded automatically by the software. Patients with IRD completed a modified Visual Function Questionnaire (VFQ).

Results

Normally sighted subjects identified approximately 50% of the obstacles at the dimmest course luminance. Except for two patients with IRD with poor vision, all patients were able to complete the test, although they required brighter (by >2 log units) luminances to identify 50% of the obstacles. In a single-luminance screening test in which normal subjects detected at least eight of nine objects, most patients with IRD underperformed; their performance related to disease severity, as measured by visual acuity, kinetic visual field extent, and VFQ scores. Test-retest differences in object detection were similar to the differences between the two eyes (±2 SD = ±2 objects).

Conclusions

This VR-O&M test was able to distinguish subjects with IRDs from normal subjects reliably and reproducibly.

Translational Relevance

This easily implemented, flexible, and objectively scored VR-O&M test promises to become a useful tool to assess the impact that IRDs and their treatments have on functional vision.

The future of sensory substitution, addition, and expansion via haptic devices

Haptic devices use the sense of touch to transmit information to the nervous system. As an example, a sound-to-touch device processes auditory information and sends it to the brain via patterns of vibration on the skin for people who have lost hearing. We here summarize the current directions of such research and draw upon examples in industry and academia. Such devices can be used for sensory substitution (replacing a lost sense, such as hearing or vision), sensory expansion (widening an existing sensory experience, such as detecting electromagnetic radiation outside the visible light spectrum), and sensory addition (providing a novel sense, such as magnetoreception). We review the relevant literature, the current status, and possible directions for the future of sensory manipulation using non-invasive haptic devices.

Preparing participants for the use of the tongue visual sensory substitution device

PURPOSE

Visual sensory substitution devices (SSDs) convey visual information to a blind person through another sensory modality. Using a visual SSD in various daily activities requires training prior to use the device independently. Yet, there is limited literature about procedures and outcomes of the training conducted for preparing users for practical use of SSDs in daily activities.

METHODS

We trained 29 blind adults (9 with congenital and 20 with acquired blindness) in the use of a commercially available electro-tactile SSD, BrainPort. We describe a structured training protocol adapted from the previous studies, responses of participants, and we present retrospective qualitative data on the progress of participants during the training.

RESULTS

The length of the training was not a critical factor in reaching an advanced stage. Though performance in the first two sessions seems to be a good indicator of participants' ability to progress in the training protocol, there are large individual differences in how far and how fast each participant can progress in the training protocol. There are differences between congenital blind users and those blinded later in life.

CONCLUSIONS

The information on the training progression would be of interest to researchers preparing studies, and to eye care professionals, who may advise patients to use SSDs.IMPLICATIONS FOR REHABILITATIONThere are large individual differences in how far and how fast each participant can learn to use a visual-to-tactile sensory substitution device for a variety of tasks.Recognition is mainly achieved through top-down processing with prior knowledge about the possible responses. Therefore, the generalizability is still questionable.Users develop different strategies in order to succeed in training tasks.

Wearable System to Guide Crosswalk Navigation for People With Visual Impairment

Independent travelling is a significant challenge for visually impaired people in urban settings. Traditional and widely used aids such as guide dogs and long canes provide basic guidance and obstacle avoidance but are not sufficient for complex situations such as street crossing. We propose a new wearable system that can safely guide a user with visual impairment at a signalized crosswalk. Safe street crossing is an important element of fully independent travelling for people who are blind or visually impaired (BVI), but street crossing is challenging for BVI because it involves several steps reliant on vision, including scene understanding, localization, object detection, path planning, and path following. Street crossing also requires timely completion. Prior solutions for guiding BVI in crosswalks have focused on either detection of crosswalks or classifying crosswalks signs. In this paper, we demonstrate a system that performs all the functions necessary to safely guide BVI at a signalized crosswalk. Our system utilizes prior maps, similar to how autonomous vehicles are guided. The hardware components are lightweight such that they can be wearable and mobile, and all are commercially available. The system operates in real-time. Computer vision algorithms (Orbslam2) localize the user in the map and orient them to the crosswalk. The state of the crosswalk signal (don’t walk or walk) is detected (using a convolutional neural network), the user is notified (via verbal instructions) when it is safe to cross, and the user is guided (via verbal instructions) along a path towards a destination on the prior map. The system continually updates user position relative to the path and corrects the user’s trajectory with simple verbal commands. We demonstrate the system functionality in three BVI participants. With brief training, all three were able to use the system to successfully navigate a crosswalk in a safe manner.

Brain-Machine Interfaces to Assist the Blind

The loss or absence of vision is probably one of the most incapacitating events that can befall a human being. The importance of vision for humans is also reflected in brain anatomy as approximately one third of the human brain is devoted to vision. It is therefore unsurprising that throughout history many attempts have been undertaken to develop devices aiming at substituting for a missing visual capacity. In this review, we present two concepts that have been prevalent over the last two decades. The first concept is sensory substitution, which refers to the use of another sensory modality to perform a task that is normally primarily sub-served by the lost sense. The second concept is cross-modal plasticity, which occurs when loss of input in one sensory modality leads to reorganization in brain representation of other sensory modalities. Both phenomena are training-dependent. We also briefly describe the history of blindness from ancient times to modernity, and then proceed to address the means that have been used to help blind individuals, with an emphasis on modern technologies, invasive (various type of surgical implants) and non-invasive devices. With the advent of brain imaging, it has become possible to peer into the neural substrates of sensory substitution and highlight the magnitude of the plastic processes that lead to a rewired brain. Finally, we will address the important question of the value and practicality of the available technologies and future directions.

Sensory augmentation to aid training with retinal prostheses

OBJECTIVE

Retinal prosthesis recipients require rehabilitative training to learn the non-intuitive nature of prosthetic 'phosphene vision'. This study investigated whether the addition of auditory cues, using The vOICe sensory substitution device (SSD), could improve functional performance with simulated phosphene vision.

APPROACH

Forty normally sighted subjects completed two visual tasks under three conditions. The phosphene condition converted the image to simulated phosphenes displayed on a virtual reality headset. The SSD condition provided auditory information via stereo headphones, translating the image into sound. Horizontal information was encoded as stereo timing differences between ears, vertical information as pitch, and pixel intensity as audio intensity. The third condition combined phosphenes and SSD. Tasks comprised light localisation from the Basic Assessment of Light and Motion (BaLM) and the Tumbling-E from the Freiburg Acuity and Contrast Test (FrACT). To examine learning effects, twenty of the forty subjects received SSD training prior to assessment.

MAIN RESULTS

Combining phosphenes with auditory SSD provided better light localisation accuracy than either phosphenes or SSD alone, suggesting a compound benefit of integrating modalities. Although response times for SSD-only were significantly longer than all other conditions, combined condition response times were as fast as phosphene-only, highlighting that audio-visual integration provided both response time and accuracy benefits. Prior SSD training provided a benefit to localisation accuracy and speed in SSD-only (as expected) and Combined conditions compared to untrained SSD-only. Integration of the two modalities did not improve spatial resolution task performance, with resolution limited to that of the higher resolution modality (SSD).

SIGNIFICANCE

Combining phosphene (visual) and SSD (auditory) modalities was effective even without SSD training and led to an improvement in light localisation accuracy and response times. Spatial resolution performance was dominated by auditory SSD. The results suggest there may be a benefit to including auditory cues when training vision prosthesis recipients.

Visual-to-auditory sensory substitution alters language asymmetry in both sighted novices and experienced visually impaired users

Visual-to-auditory sensory substitution devices (SSDs) provide improved access to the visual environment for the visually impaired by converting images into auditory information. Research is lacking on the mechanisms involved in processing data that is perceived through one sensory modality, but directly associated with a source in a different sensory modality. This is important because SSDs that use auditory displays could involve binaural presentation requiring both ear canals, or monaural presentation requiring only one - but which ear would be ideal? SSDs may be similar to reading, as an image (printed word) is converted into sound (when read aloud). Reading, and language more generally, are typically lateralised to the left cerebral hemisphere. Yet, unlike symbolic written language, SSDs convert images to sound based on visuospatial properties, with the right cerebral hemisphere potentially having a role in processing such visuospatial data. Here we investigated whether there is a hemispheric bias in the processing of visual-to-auditory sensory substitution information and whether that varies as a function of experience and visual ability. We assessed the lateralization of auditory processing with two tests: a standard dichotic listening test and a novel dichotic listening test created using the auditory information produced by an SSD, The vOICe. Participants were tested either in the lab or online with the same stimuli. We did not find a hemispheric bias in the processing of visual-to-auditory information in visually impaired, experienced vOICe users. Further, we did not find any difference between visually impaired, experienced vOICe users and sighted novices in the hemispheric lateralization of visual-to-auditory information processing. Although standard dichotic listening is lateralised to the left hemisphere, the auditory processing of images in SSDs is bilateral, possibly due to the increased influence of right hemisphere processing. Auditory SSDs might therefore be equally effective with presentation to either ear if a monaural, rather than binaural, presentation were necessary.

The Use of Electrotherapeutics in Ophthalmology

PURPOSE

To present a perspective on the use of electrotherapeutics in the history of ophthalmology along with the development of novel contemporary ophthalmic instrumentation.

DESIGN

Perspective study.

METHODS

We reviewed historical journals, articles, and books discussing the use of electricity and electrotherapeutics in ophthalmology.

RESULTS

Electrotherapeutic applications have been researched and used to treat ocular diseases as far back as the 18th century. By the 20th century, research in electrotherapeutics in ophthalmology had caught the eye of Edward Jackson, the first president of the American Academy of Ophthalmology and Otolaryngology and first editor of the present (third) series American Journal of Ophthalmology. Edward Jackson published an extensive review on this topic and reported a variety of modalities used to treat ocular diseases.

CONCLUSIONS

While many early therapeutic uses of electricity did not produce effective and replicable results, studies on electrical stimulation of the eye provided the foundation for the development of clinically significant vision enhancing and restoring instrumentation.

Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision

Purpose

Efficacy of current visual prostheses in object recognition is limited. Among various limitations to be addressed, such as low resolution and low dynamic range, here we focus on reducing the impact of background clutter on object recognition. We have proposed the use of motion parallax via head-mounted camera lateral scanning and computationally stabilizing the object of interest (OI) to support neural background decluttering. Simulations in head-mounted displays (HMD), mimicking the proposed effect, were used to test object recognition in normally sighted subjects.

Methods

Images (24° field of view) were captured from multiple viewpoints and presented at a low resolution (20 × 20). All viewpoints were centered on the OI. Experimental conditions (2 × 3) included clutter (with or without) × head scanning (single viewpoint, 9 coherent viewpoints corresponding to subjects' head positions, and 9 randomly associated viewpoints). Subjects used lateral head movements to view OIs in the HMD. Each object was displayed only once for each subject.

Results

The median recognition rate without clutter was 40% for all head scanning conditions. Performance with synthetic background clutter dropped to 10% in the static condition, but it was improved to 20% with the coherent and random head scanning (corrected P = 0.005 and P = 0.049, respectively).

Conclusions

Background decluttering using motion parallax cues but not the coherent multiple views of the OI improved object recognition in low-resolution images. The improvement did not fully eliminate the impact of background.

Translational Relevance

Motion parallax is an effective but incomplete decluttering solution for object recognition with visual prostheses.

Word recognition: re-thinking prosthetic vision evaluation

OBJECTIVE

Evaluations of vision prostheses and sensory substitution devices have frequently relied on repeated training and then testing with the same small set of items. These multiple forced-choice tasks produced above chance performance in blind users, but it is unclear if the observed performance represents restoration of vision that transfers to novel, untrained items.

APPROACH

Here, we tested the generalizability of the forced-choice paradigm on discrimination of low-resolution word images. Extensive visual training was conducted with the same 10 words used in previous BrainPort tongue stimulation studies. The performance on these 10 words and an additional 50 words was measured before and after the training sessions.

MAIN RESULTS

The results revealed minimal performance improvement with the untrained words, demonstrating instead pattern discrimination limited mostly to the trained words.

SIGNIFICANCE

These findings highlight the need to reconsider current evaluation practices, in particular, the use of forced-choice paradigms with a few highly trained items. While appropriate for measuring the performance thresholds in acuity or contrast sensitivity of a functioning visual system, performance on such tasks cannot be taken to indicate restored spatial pattern vision.

Designing sensory-substitution devices: Principles, pitfalls and potential1

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.

Brain mechanisms of Change in Addictions Treatment: Models, Methods, and Emerging Findings

Increased understanding of "how" and "for whom" treatment works at the level of the brain has potential to transform addictions treatment through the development of innovative neuroscience-informed interventions. The 2015 Science of Change meeting bridged the fields of neuroscience and psychotherapy research to identify brain mechanisms of behavior change that are "common" across therapies, and "specific" to distinct behavioral interventions. Conceptual models of brain mechanisms underlying effects of Cognitive Behavioral Therapy, mindfulness interventions, and Motivational Interviewing were discussed. Presentations covered methods for integrating neuroimaging into psychotherapy research, and novel analytic approaches. Effects of heavy substance use on the brain, and recovery of brain functioning with sustained abstinence, which may be facilitated by cognitive training, were reviewed. Neuroimaging provides powerful tools for determining brain mechanisms underlying psychotherapy and medication effects, predicting and monitoring outcomes, developing novel interventions that target specific brain circuits, and identifying for whom an intervention will be effective.

Electronic approaches to restoration of sight

Retinal prostheses are a promising means for restoring sight to patients blinded by the gradual atrophy of photoreceptors due to retinal degeneration. They are designed to reintroduce information into the visual system by electrically stimulating surviving neurons in the retina. This review outlines the concepts and technologies behind two major approaches to retinal prosthetics: epiretinal and subretinal. We describe how the visual system responds to electrical stimulation. We highlight major differences between direct encoding of the retinal output with epiretinal stimulation, and network-mediated response with subretinal stimulation. We summarize results of pre-clinical evaluation of prosthetic visual functions in- and ex vivo, as well as the outcomes of current clinical trials of various retinal implants. We also briefly review alternative, non-electronic, approaches to restoration of sight to the blind, and conclude by suggesting some perspectives for future advancement in the field.

Improving training for sensory augmentation using the science of expertise

Sensory substitution and augmentation devices (SSADs) allow users to perceive information about their environment that is usually beyond their sensory capabilities. Despite an extensive history, SSADs are arguably not used to their fullest, both as assistive technology for people with sensory impairment or as research tools in the psychology and neuroscience of sensory perception. Studies of the non-use of other assistive technologies suggest one factor is the balance of benefits gained against the costs incurred. We argue that improving the learning experience would improve this balance, suggest three ways in which it can be improved by leveraging existing cognitive science findings on expertise and skill development, and acknowledge limitations and relevant concerns. We encourage the systematic evaluation of learning programs, and suggest that a more effective learning process for SSADs could reduce the barrier to uptake and allow users to reach higher levels of overall capacity.

Shape Discrimination Using the Tongue: Implications for a Visual-to-Tactile Sensory Substitution Device

Sensory substitution devices have the potential to provide individuals with visual impairments with more information about their environments, which may help them recognize objects and achieve more independence in their daily lives. However, many of these devices may require extensive training and might be limited in the amount of information that they can convey. We tested the effectiveness and assessed some of the limitations of the BrainPort device, which provides stimulation through a 20 × 20 electrode grid array on the tongue. Across five experiments, including one with blind individuals, we found that subjects were unable to accurately discriminate between simple shapes as well as different line orientations that were briefly presented on the tongue, even after 300 trials of practice with the device. These experiments indicate that such a minimal training regimen with the BrainPort is not sufficient for object recognition, raising serious concerns about the usability of this device without extensive training.

Top-down influence on the visual cortex of the blind during sensory substitution

Visual sensory substitution devices provide a non-surgical and flexible approach to vision rehabilitation in the blind. These devices convert images taken by a camera into cross-modal sensory signals that are presented as a surrogate for direct visual input. While previous work has demonstrated that the visual cortex of blind subjects is recruited during sensory substitution, the cognitive basis of this activation remains incompletely understood. To test the hypothesis that top-down input provides a significant contribution to this activation, we performed functional MRI scanning in 11 blind (7 acquired and 4 congenital) and 11 sighted subjects under two conditions: passive listening of image-encoded soundscapes before sensory substitution training and active interpretation of the same auditory sensory substitution signals after a 10-minute training session. We found that the modulation of visual cortex activity due to active interpretation was significantly stronger in the blind over sighted subjects. In addition, congenitally blind subjects showed stronger task-induced modulation in the visual cortex than acquired blind subjects. In a parallel experiment, we scanned 18 blind (11 acquired and 7 congenital) and 18 sighted subjects at rest to investigate alterations in functional connectivity due to visual deprivation. The results demonstrated that visual cortex connectivity of the blind shifted away from sensory networks and toward known areas of top-down input. Taken together, our data support the model of the brain, including the visual system, as a highly flexible task-based and not sensory-based machine.