Gaze Compensation as a Technique for Improving Hand-Eye Coordination in Prosthetic Vision

The Latest Research Progress on Bionic Artificial Hands: A Systematic Review

Bionic prosthetic hands hold the potential to replicate the functionality of human hands. The use of bionic limbs can assist amputees in performing everyday activities. This article systematically reviews the research progress on bionic prostheses, with a focus on control mechanisms, sensory feedback integration, and mechanical design innovations. It emphasizes the use of bioelectrical signals, such as electromyography (EMG), for prosthetic control and discusses the application of machine learning algorithms to enhance the accuracy of gesture recognition. Additionally, the paper explores advancements in sensory feedback technologies, including tactile, visual, and auditory modalities, which enhance user interaction by providing essential environmental feedback. The mechanical design of prosthetic hands is also examined, with particular attention to achieving a balance between dexterity, weight, and durability. Our contribution consists of compiling current research trends and identifying key areas for future development, including the enhancement of control system integration and improving the aesthetic and functional resemblance of prostheses to natural limbs. This work aims to inform and inspire ongoing research that seeks to refine the utility and accessibility of prosthetic hands for amputees, emphasizing user-centric innovations.

Gaze-contingent processing improves mobility, scene recognition and visual search in simulated head-steered prosthetic vision

Objective.The enabling technology of visual prosthetics for the blind is making rapid progress. However, there are still uncertainties regarding the functional outcomes, which can depend on many design choices in the development. In visual prostheses with a head-mounted camera, a particularly challenging question is how to deal with the gaze-locked visual percept associated with spatial updating conflicts in the brain. The current study investigates a recently proposed compensation strategy based on gaze-contingent image processing with eye-tracking. Gaze-contingent processing is expected to reinforce natural-like visual scanning and reestablished spatial updating based on eye movements. The beneficial effects remain to be investigated for daily life activities in complex visual environments.Approach.The current study evaluates the benefits of gaze-contingent processing versus gaze-locked and gaze-ignored simulations in the context of mobility, scene recognition and visual search, using a virtual reality simulated prosthetic vision paradigm with sighted subjects.Main results.Compared to gaze-locked vision, gaze-contingent processing was consistently found to improve the speed in all experimental tasks, as well as the subjective quality of vision. Similar or further improvements were found in a control condition that ignores gaze-dependent effects, a simulation that is unattainable in the clinical reality.Significance.Our results suggest that gaze-locked vision and spatial updating conflicts can be debilitating for complex visually-guided activities of daily living such as mobility and orientation. Therefore, for prospective users of head-steered prostheses with an unimpaired oculomotor system, the inclusion of a compensatory eye-tracking system is strongly endorsed.

Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

A novel simulation paradigm utilising MRI-derived phosphene maps for cortical prosthetic vision

Objective.We developed a realistic simulation paradigm for cortical prosthetic vision and investigated whether we can improve visual performance using a novel clustering algorithm.Approach.Cortical visual prostheses have been developed to restore sight by stimulating the visual cortex. To investigate the visual experience, previous studies have used uniform phosphene maps, which may not accurately capture generated phosphene map distributions of implant recipients. The current simulation paradigm was based on the Human Connectome Project retinotopy dataset and the placement of implants on the cortices from magnetic resonance imaging scans. Five unique retinotopic maps were derived using this method. To improve performance on these retinotopic maps, we enabled head scanning and a density-based clustering algorithm was then used to relocate centroids of visual stimuli. The impact of these improvements on visual detection performance was tested. Using spatially evenly distributed maps as a control, we recruited ten subjects and evaluated their performance across five sessions on the Berkeley Rudimentary Visual Acuity test and the object recognition task.Main results.Performance on control maps is significantly better than on retinotopic maps in both tasks. Both head scanning and the clustering algorithm showed the potential of improving visual ability across multiple sessions in the object recognition task.Significance.The current paradigm is the first that simulates the experience of cortical prosthetic vision based on brain scans and implant placement, which captures the spatial distribution of phosphenes more realistically. Utilisation of evenly distributed maps may overestimate the performance that visual prosthetics can restore. This simulation paradigm could be used in clinical practice when making plans for where best to implant cortical visual prostheses.

A systematic review of extended reality (XR) for understanding and augmenting vision loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user's eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

Estimating Phosphene Locations Using Eye Movements of Suprachoroidal Retinal Prosthesis Users

Purpose

Accurate mapping of phosphene locations from visual prostheses is vital to encode spatial information. This process may involve the subject pointing to evoked phosphene locations with their finger. Here, we demonstrate phosphene mapping for a retinal implant using eye movements and compare it with retinotopic electrode positions and previous results using conventional finger-based mapping.

Methods

Three suprachoroidal retinal implant recipients (NCT03406416) indicated the spatial position of phosphenes. Electrodes were stimulated individually, and the subjects moved their finger (finger based) or their eyes (gaze based) to the perceived phosphene location. The distortion of the measured phosphene locations from the expected locations (retinotopic electrode locations) was characterized with Procrustes analysis.

Results

The finger-based phosphene locations were compressed spatially relative to the expected locations all three subjects, but preserved the general retinotopic arrangement (scale factors ranged from 0.37 to 0.83). In two subjects, the gaze-based phosphene locations were similar to the expected locations (scale factors of 0.72 and 0.99). For the third subject, there was no apparent relationship between gaze-based phosphene locations and electrode locations (scale factor of 0.07).

Conclusions

Gaze-based phosphene mapping was achievable in two of three tested retinal prosthesis subjects and their derived phosphene maps correlated well with the retinotopic electrode layout. A third subject could not produce a coherent gaze-based phosphene map, but this may have revealed that their phosphenes were indistinct spatially.

Translational Relevance

Gaze-based phosphene mapping is a viable alternative to conventional finger-based mapping, but may not be suitable for all subjects.

Immersive Virtual Reality Simulations of Bionic Vision

Bionic vision uses neuroprostheses to restore useful vision to people living with incurable blindness. However, a major outstanding challenge is predicting what people "see" when they use their devices. The limited field of view of current devices necessitates head movements to scan the scene, which is difficult to simulate on a computer screen. In addition, many computational models of bionic vision lack biological realism. To address these challenges, we present VR-SPV, an open-source virtual reality toolbox for simulated prosthetic vision that uses a psychophysically validated computational model to allow sighted participants to "see through the eyes" of a bionic eye user. To demonstrate its utility, we systematically evaluated how clinically reported visual distortions affect performance in a letter recognition and an immersive obstacle avoidance task. Our results highlight the importance of using an appropriate phosphene model when predicting visual outcomes for bionic vision.

A Second-Generation (44-Channel) Suprachoroidal Retinal Prosthesis: Interim Clinical Trial Results

Purpose

To report the initial safety and efficacy results of a second-generation (44-channel) suprachoroidal retinal prosthesis at 56 weeks after device activation.

Methods

Four subjects, with advanced retinitis pigmentosa and bare-light perception only, enrolled in a phase II trial (NCT03406416). A 44-channel electrode array was implanted in a suprachoroidal pocket. Device stability, efficacy, and adverse events were investigated at 12-week intervals.

Results

All four subjects were implanted successfully and there were no device-related serious adverse events. Color fundus photography indicated a mild postoperative subretinal hemorrhage in two recipients, which cleared spontaneously within 2 weeks. Optical coherence tomography confirmed device stability and position under the macula. Screen-based localization accuracy was significantly better for all subjects with device on versus device off. Two subjects were significantly better with the device on in a motion discrimination task at 7, 15, and 30°/s and in a spatial discrimination task at 0.033 cycles per degree. All subjects were more accurate with the device on than device off at walking toward a target on a modified door task, localizing and touching tabletop objects, and detecting obstacles in an obstacle avoidance task. A positive effect of the implant on subjects' daily lives was confirmed by an orientation and mobility assessor and subject self-report.

Conclusions

These interim study data demonstrate that the suprachoroidal prosthesis is safe and provides significant improvements in functional vision, activities of daily living, and observer-rated quality of life.

Translational Relevance

A suprachoroidal prosthesis can provide clinically useful artificial vision while maintaining a safe surgical profile.

Full gaze contingency provides better reading performance than head steering alone in a simulation of prosthetic vision

The visual pathway is retinotopically organized and sensitive to gaze position, leading us to hypothesize that subjects using visual prostheses incorporating eye position would perform better on perceptual tasks than with devices that are merely head-steered. We had sighted subjects read sentences from the MNREAD corpus through a simulation of artificial vision under conditions of full gaze compensation, and head-steered viewing. With 2000 simulated phosphenes, subjects (n = 23) were immediately able to read under full gaze compensation and were assessed at an equivalent visual acuity of 1.0 logMAR, but were nearly unable to perform the task under head-steered viewing. At the largest font size tested, 1.4 logMAR, subjects read at 59 WPM (50% of normal speed) with 100% accuracy under the full-gaze condition, but at 0.7 WPM (under 1% of normal) with below 15% accuracy under head-steering. We conclude that gaze-compensated prostheses are likely to produce considerably better patient outcomes than those not incorporating eye movements.

Advances in Neuroscience, Not Devices, Will Determine the Effectiveness of Visual Prostheses

Background: Innovations in engineering and neuroscience have enabled the development of sophisticated visual prosthetic devices. In clinical trials, these devices have provided visual acuities as high as 20/460, enabled coarse navigation, and even allowed for reading of short words. However, long-term commercial viability arguably rests on attaining even better vision and more definitive improvements in tasks of daily living and quality of life. Purpose: Here we review technological and biological obstacles in the implementation of visual prosthetics. Conclusions: Research in the visual prosthetic field has tackled significant technical challenges, including biocompatibility, signal spread through neural tissue, and inadvertent activation of passing axons; however, significant gaps in knowledge remain in the realm of neuroscience, including the neural code of vision and visual plasticity. We assert that further optimization of prosthetic devices alone will not provide markedly improved visual outcomes without significant advances in our understanding of neuroscience.

Oculomotor Responses to Dynamic Stimuli in a 44-Channel Suprachoroidal Retinal Prosthesis

Purpose

To investigate oculomotor behavior in response to dynamic stimuli in retinal implant recipients.

Methods

Three suprachoroidal retinal implant recipients performed a four-alternative forced-choice motion discrimination task over six sessions longitudinally. Stimuli were a single white bar (“moving bar”) or a series of white bars (“moving grating”) sweeping left, right, up, or down across a 42″ monitor. Performance was compared with normal video processing and scrambled video processing (randomized image-to-electrode mapping to disrupt spatiotemporal structure). Eye and head movement was monitored throughout the task.

Results

Two subjects had diminished performance with scrambling, suggesting retinotopic discrimination was used in the normal condition and made smooth pursuit eye movements congruent to the moving bar stimulus direction. These two subjects also made stimulus-related eye movements resembling optokinetic reflex (OKR) for moving grating stimuli, but the movement was incongruent with stimulus direction. The third subject was less adept at the task, appeared primarily reliant on head position cues (head movements were congruent to stimulus direction), and did not exhibit retinotopic discrimination and associated eye movements.

Conclusions

Our observation of smooth pursuit indicates residual functionality of cortical direction-selective circuits and implies a more naturalistic perception of motion than expected. A distorted OKR implies improper functionality of retinal direction-selective circuits, possibly due to retinal remodeling or the non-selective nature of the electrical stimulation.

Translational Relevance

Retinal implant users can make naturalistic eye movements in response to moving stimuli, highlighting the potential for eye tracker feedback to improve perceptual localization and image stabilization in camera-based visual prostheses.

An update on retinal prostheses

Retinal prostheses are designed to restore a basic sense of sight to people with profound vision loss. They require a relatively intact posterior visual pathway (optic nerve, lateral geniculate nucleus and visual cortex). Retinal implants are options for people with severe stages of retinal degenerative disease such as retinitis pigmentosa and age-related macular degeneration. There have now been three regulatory-approved retinal prostheses. Over five hundred patients have been implanted globally over the past 15 years. Devices generally provide an improved ability to localize high-contrast objects, navigate, and perform basic orientation tasks. Adverse events have included conjunctival erosion, retinal detachment, loss of light perception, and the need for revision surgery, but are rare. There are also specific device risks, including overstimulation (which could cause damage to the retina) or delamination of implanted components, but these are very unlikely. Current challenges include how to improve visual acuity, enlarge the field-of-view, and reduce a complex visual scene to its most salient components through image processing. This review encompasses the work of over 40 individual research groups who have built devices, developed stimulation strategies, or investigated the basic physiology underpinning retinal prostheses. Current technologies are summarized, along with future challenges that face the field.

Head and Gaze Behavior in Retinitis Pigmentosa

Purpose

Peripheral visual field loss (PVFL) due to retinitis pigmentosa (RP) decreases saccades to areas of visual defect, leading to a habitually confined range of eye movement. We investigated the relative contributions of head and eye movement in RP patients and normal-sighted controls to determine whether this reduced eye movement is offset by increased head movement.

Methods

Eye-head coordination was examined in 18 early-moderate RP patients, 4 late-stage RP patients, and 19 normal-sighted controls. Three metrics were extracted: the extent of eye, head, and total gaze (eye+head) movement while viewing a naturalistic scene; head gain, the ratio of head movement to total gaze movement during smooth pursuit; and the customary oculomotor range (COMR), the orbital range within which the eye is preferentially maintained during a pro-saccade task.

Results

The late-stage RP group had minimal gaze movement and could not discern the naturalistic scene. Variance in head position in early-moderate RP was significantly greater than in controls, whereas variance in total gaze was similar. Head gain was greater in early-moderate RP than in controls, whereas COMR was smaller. Across groups, visual field extent was negatively correlated with head gain and positively correlated with COMR. Accounting for age effects, these results demonstrate increased head movement at the expense of eye movement in participants with PVFL.

Conclusions

RP is associated with an increased propensity for head movement during gaze shifts, and the magnitude of this effect is dependent on the severity of visual field loss.

The influence of visual field position induced by a retinal prosthesis simulator on mobility

PURPOSE

Our aim is to develop a new generation of suprachoroidal-transretinal stimulation (STS) retinal prosthesis using a dual-stimulating electrode array to enlarge the visual field. In the present study, we aimed to examine how position and size of the visual field-created by a retinal prosthesis simulator-influenced mobility.

METHODS

Twelve healthy subjects wore retinal prosthesis simulators. Images captured by a web camera attached to a head-mounted display (HMD) were processed by a computer and displayed on the HMD. Three types of artificial visual fields-designed to imitate phosphenes-obtained by a single (5 × 5 electrodes; visual angle, 15°) or dual (5 × 5 electrodes ×2; visual angle, 30°) electrode array were created. Visual field (VF)1 is an inferior visual field, which corresponds to a dual-electrode array implanted in the superior hemisphere. VF2 is a superior visual field, which corresponds to a single-electrode array implanted in the inferior hemisphere. VF3 is a superior visual field, which corresponds to a dual-electrode array implanted in the inferior hemisphere. In each type of artificial visual field, a natural circular visual field (visual angle, 5°) which imitated the vision of patients with advanced retinitis pigmentosa existed at the center. Subjects were instructed to walk along a black carpet (6 m long × 2.2 m wide) without stepping on attached white circular obstacles. Each obstacle was 20 cm in diameter, and obstacles were installed at 40-cm intervals. We measured the number of footsteps on the obstacles, the time taken to complete the obstacle course, and the extent of head movement to scan the area (head-scanning). We then compared the results recorded from these 3 types of artificial visual field.

RESULTS

The number of footsteps on obstacles was lowest in VF3 (One-way ANOVA; P = 0.028, Fisher's LSD; VF 1 versus 3 P = 0.039, 2 versus 3 P = 0.012). No significant difference was observed for the time to complete the obstacle course or the extent of head movement between the 3 visual fields.

CONCLUSION

The superior and wide visual field (VF3) obtained by the retinal prosthesis simulator resulted in better mobility performance than the other visual fields.