Eye Movement Compensation and Spatial Updating in Visual Prosthetics: Mechanisms, Limitations and Future Directions

Gaze-contingent processing improves mobility, scene recognition and visual search in simulated head-steered prosthetic vision

Objective.The enabling technology of visual prosthetics for the blind is making rapid progress. However, there are still uncertainties regarding the functional outcomes, which can depend on many design choices in the development. In visual prostheses with a head-mounted camera, a particularly challenging question is how to deal with the gaze-locked visual percept associated with spatial updating conflicts in the brain. The current study investigates a recently proposed compensation strategy based on gaze-contingent image processing with eye-tracking. Gaze-contingent processing is expected to reinforce natural-like visual scanning and reestablished spatial updating based on eye movements. The beneficial effects remain to be investigated for daily life activities in complex visual environments.Approach.The current study evaluates the benefits of gaze-contingent processing versus gaze-locked and gaze-ignored simulations in the context of mobility, scene recognition and visual search, using a virtual reality simulated prosthetic vision paradigm with sighted subjects.Main results.Compared to gaze-locked vision, gaze-contingent processing was consistently found to improve the speed in all experimental tasks, as well as the subjective quality of vision. Similar or further improvements were found in a control condition that ignores gaze-dependent effects, a simulation that is unattainable in the clinical reality.Significance.Our results suggest that gaze-locked vision and spatial updating conflicts can be debilitating for complex visually-guided activities of daily living such as mobility and orientation. Therefore, for prospective users of head-steered prostheses with an unimpaired oculomotor system, the inclusion of a compensatory eye-tracking system is strongly endorsed.

Nanoparticle-based optical interfaces for retinal neuromodulation: a review

Degeneration of photoreceptors in the retina is a leading cause of blindness, but commonly leaves the retinal ganglion cells (RGCs) and/or bipolar cells extant. Consequently, these cells are an attractive target for the invasive electrical implants colloquially known as "bionic eyes." However, after more than two decades of concerted effort, interfaces based on conventional electrical stimulation approaches have delivered limited efficacy, primarily due to the current spread in retinal tissue, which precludes high-acuity vision. The ideal prosthetic solution would be less invasive, provide single-cell resolution and an ability to differentiate between different cell types. Nanoparticle-mediated approaches can address some of these requirements, with particular attention being directed at light-sensitive nanoparticles that can be accessed via the intrinsic optics of the eye. Here we survey the available known nanoparticle-based optical transduction mechanisms that can be exploited for neuromodulation. We review the rapid progress in the field, together with outstanding challenges that must be addressed to translate these techniques to clinical practice. In particular, successful translation will likely require efficient delivery of nanoparticles to stable and precisely defined locations in the retinal tissues. Therefore, we also emphasize the current literature relating to the pharmacokinetics of nanoparticles in the eye. While considerable challenges remain to be overcome, progress to date shows great potential for nanoparticle-based interfaces to revolutionize the field of visual prostheses.

Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

Estimating Phosphene Locations Using Eye Movements of Suprachoroidal Retinal Prosthesis Users

Purpose

Accurate mapping of phosphene locations from visual prostheses is vital to encode spatial information. This process may involve the subject pointing to evoked phosphene locations with their finger. Here, we demonstrate phosphene mapping for a retinal implant using eye movements and compare it with retinotopic electrode positions and previous results using conventional finger-based mapping.

Methods

Three suprachoroidal retinal implant recipients (NCT03406416) indicated the spatial position of phosphenes. Electrodes were stimulated individually, and the subjects moved their finger (finger based) or their eyes (gaze based) to the perceived phosphene location. The distortion of the measured phosphene locations from the expected locations (retinotopic electrode locations) was characterized with Procrustes analysis.

Results

The finger-based phosphene locations were compressed spatially relative to the expected locations all three subjects, but preserved the general retinotopic arrangement (scale factors ranged from 0.37 to 0.83). In two subjects, the gaze-based phosphene locations were similar to the expected locations (scale factors of 0.72 and 0.99). For the third subject, there was no apparent relationship between gaze-based phosphene locations and electrode locations (scale factor of 0.07).

Conclusions

Gaze-based phosphene mapping was achievable in two of three tested retinal prosthesis subjects and their derived phosphene maps correlated well with the retinotopic electrode layout. A third subject could not produce a coherent gaze-based phosphene map, but this may have revealed that their phosphenes were indistinct spatially.

Translational Relevance

Gaze-based phosphene mapping is a viable alternative to conventional finger-based mapping, but may not be suitable for all subjects.

Reading text works better than watching videos to improve acuity in a simulation of artificial vision

Simulated artificial vision is used in visual prosthesis design to answer questions about device usability. We previously reported a striking increase in equivalent visual acuity with daily use of a simulation of artificial vision in an active task, reading sentences, that required high levels of subject engagement, but passive activities are more likely to dominate post-implant experience. Here, we investigated the longitudinal effects of a passive task, watching videos. Eight subjects used a simulation of a thalamic visual prosthesis with 1000 phosphenes to watch 23 episodes of classic American television in daily, 25-min sessions, for a period of 1 month with interspersed reading tests that quantified reading accuracy and reading speed. For reading accuracy, we found similar dynamics to the early part of the learning process in our previous report, here leading to an improvement in visual acuity of 0.15 ± 0.05 logMAR. For reading speed, however, no change was apparent by the end of training. We found that single reading sessions drove about twice the improvement in acuity of single video sessions despite being only half as long. We conclude that while passive viewing tasks may prove useful for post-implant rehabilitation, active tasks are likely to be preferable.

Virtual reality validation of naturalistic modulation strategies to counteract fading in retinal stimulation

OBJECTIVE

Temporal resolution is a key challenge in artificial vision. Several prosthetic approaches are limited by the perceptual fading of evoked phosphenes upon repeated stimulation from the same electrode. Therefore, implanted patients are forced to perform active scanning, via head movements, to refresh the visual field viewed by the camera. However, active scanning is a draining task, and it is crucial to find compensatory strategies to reduce it.

APPROACH

To address this question, we implemented perceptual fading in simulated prosthetic vision using virtual reality. Then, we quantified the effect of fading on two indicators: the time to complete a reading task and the head rotation during the task. We also tested if stimulation strategies previously proposed to increase the persistence of responses in retinal ganglion cells to electrical stimulation could improve these indicators.

MAIN RESULTS

This study shows that stimulation strategies based on interrupted pulse trains and randomisation of the pulse duration allows significant reduction of both the time to complete the task and the head rotation during the task.

SIGNIFICANCE

The stimulation strategy used in retinal implants is crucial to counteract perceptual fading and to reduce active head scanning during prosthetic vision. In turn, less active scanning might improve the patient's comfort in artificial vision.

Modern concepts of bionic vision

Loss of vision is a pressing medical and social problem leading to profound disability, loss of ability to work, serious alterations in the psycho-emotional state, and a decline of the quality of life. When conservative or surgical treatment can not help restore vision, the use of visual prosthesis - bionic eye - can be an effective solution. This review covers the main modern approaches to the development of visual prosthetic systems. Analysis of publications revealed that there are several main approaches to visual prosthesis differing primarily by the anatomical structure targeted for stimulation in order to activate visual sensations. The most significant among them are retinal prostheses, optic nerve stimulation, and cortical visual prostheses. Currently, retinal prostheses such as ARGUS II demonstrate the most successful results, since the stimulation of the surviving neural structures of the retina is a relatively easy task, but their field of application is limited to diseases associated with pathological changes in photoreceptors. The development of cortical visual prostheses is more difficult, but in the future they may allow using more stimulation channels to obtain a more detailed visual perception. In addition, cortical visual prostheses are universal, as they do not require preservation of any structures of the visual organ, only the primary visual cortex.

A Treatise on Electrode Carrier Dislocation in Visual Prosthetic Devices

Visual implants electrically activate adjacent neurons to induce artificial perception for visual impairment patients to restore some sight. Proximity of electrode carrier to the ganglion cell has attracted careful consideration due to its implications on secure electrochemical and single-localized stimulation. In this study, we postulate a novel strategy to treat the proximity of electrode-cell. A simulation framework includes the carrier dislocation using the geometric parameters of Argus II® epiretinal electrode carrier design. Lastly, we present results on the offset angle of displacement.Clinical Relevance- This postulates a novel strategy to treat the dislocation of electrode carrier confined with a single tack.

Multi-electrode stimulation evokes consistent spatial patterns of phosphenes and improves phosphene mapping in blind subjects

BACKGROUND

Visual cortical prostheses (VCPs) have the potential to restore visual function to patients with acquired blindness. Successful implementation of VCPs requires the ability to reliably map the location of the phosphene produced by stimulation of each implanted electrode.

OBJECTIVE

To evaluate the efficacy of different approaches to phosphene mapping and propose simple improvements to mapping strategy.

METHODS

We stimulated electrodes implanted in the visual cortex of five blind and fifteen sighted patients. We tested two fixation strategies, unimanual fixation, where subjects placed a single index finger on a tactile fixation point and bimanual fixation, where subjects overlaid their right index finger over their left on the tactile point. In addition, we compared absolute mapping in which a single electrode was stimulated on each trial, and relative mapping with sequences containing stimulation of three to five phosphenes on each trial. Trial-to-trial variability present in relative mapping sequences was quantified.

RESULTS

Phosphene mapping was less precise in blind subjects than in sighted subjects (2DRMS, 16 ± 2.9° vs. 1.9 ± 0.93°; t (18) = 18, p = <0.001). Within blind subjects, bimanual fixation resulted in more consistent phosphene localization than unimanual fixation (BS1: 4.0 ± 2.6° vs. 19 ± 4.7°, t (79) = 24, p < 0.001; BS2 4.1 ± 2.0° vs. 12 ± 2.7°, t (65) = 19, p < 0.001). Multi-point relative mapping had similar baseline precision to absolute mapping (BS1: 4.7 ± 2.6° vs. 3.9 ± 2.0°; BS2: 4.1 ± 2.0° vs. 3.2 ± 1.1°) but improved significantly when trial-to-trial translational variability was removed. Although multi-point mapping methods did reveal more of the functional organization expected in early visual cortex, subjects tended to artificially regularize the spacing between phosphenes. We attempt to address this issue by fitting a standard logarithmic map to relative multi-point sequences.

CONCLUSIONS

Relative mapping methods, combined with bimanual fixation, resulted in the most precise estimates of phosphene organization. These techniques, combined with use of a standard logarithmic model of visual cortex, may provide a practical way to improve the implementation of a VCP.

Full gaze contingency provides better reading performance than head steering alone in a simulation of prosthetic vision

The visual pathway is retinotopically organized and sensitive to gaze position, leading us to hypothesize that subjects using visual prostheses incorporating eye position would perform better on perceptual tasks than with devices that are merely head-steered. We had sighted subjects read sentences from the MNREAD corpus through a simulation of artificial vision under conditions of full gaze compensation, and head-steered viewing. With 2000 simulated phosphenes, subjects (n = 23) were immediately able to read under full gaze compensation and were assessed at an equivalent visual acuity of 1.0 logMAR, but were nearly unable to perform the task under head-steered viewing. At the largest font size tested, 1.4 logMAR, subjects read at 59 WPM (50% of normal speed) with 100% accuracy under the full-gaze condition, but at 0.7 WPM (under 1% of normal) with below 15% accuracy under head-steering. We conclude that gaze-compensated prostheses are likely to produce considerably better patient outcomes than those not incorporating eye movements.

Eye movements and the perceived location of phosphenes generated by intracranial primary visual cortex stimulation in the blind

BACKGROUND

Restoring sight for the blind using electrical stimulation of the visual pathways is feasible but demands an understanding of the spatial mapping of the visual world at the site of targeted stimulation, whether in the retina, thalamus, or cortex. While a visual cortex stimulator can bypass the eye and create visual percepts, there is an inherent dissociation between this stimulation and eye movements. It is unknown whether and how robustly the brain maintains the oculomotor circuitry in patients with bare- or no-light perception.

OBJECTIVE

To critically and quantitatively evaluate the effect of eye movements have on phosphene locations elicited by cortical stimulation that bypasses the eyes in order to restore sight in blind subjects.

METHODS

The NeuroPace Responsive Neurostimulator (RNS) and the Orion visual cortical prosthesis devices were used to electrically stimulate the visual cortex of blind subjects with bare or no light perception. Eye positions were recorded synchronized with stimulation and the location of the percepts were measured using a handheld marker.

RESULTS

The locations of cortical stimulation-evoked percepts are shifted based on the eye position at the time of stimulation. Measured responses can be remapped based on measured eye positions to determine the retinotopic locations associated with the implanted electrodes, with remapped responses having variance limited by pointing error.

CONCLUSIONS

Eye movements dominate the perceived location of cortical stimulation-evoked phosphenes, even after years of blindness. By accounting for eye positions, we can mimic retinal mapping as in natural sight.

The impact of synchronous versus asynchronous electrical stimulation in artificial vision

Visual prosthesis devices designed to restore sight to the blind have been under development in the laboratory for several decades. Clinical translation continues to be challenging, due in part to gaps in our understanding of critical parameters such as how phosphenes, the electrically-generated pixels of artificial vision, can be combined to form images. In this review we explore the effects that synchronous and asynchronous electrical stimulation across multiple electrodes have in evoking phosphenes. Understanding how electrical patterns influence phosphene generation to control object binding and perception of visual form is fundamental to creation of a clinically successful prosthesis.

Optogenetic therapy: high spatiotemporal resolution and pattern discrimination compatible with vision restoration in non-human primates

Vision restoration is an ideal medical application for optogenetics, because the eye provides direct optical access to the retina for stimulation. Optogenetic therapy could be used for diseases involving photoreceptor degeneration, such as retinitis pigmentosa or age-related macular degeneration. We describe here the selection, in non-human primates, of a specific optogenetic construct currently tested in a clinical trial. We used the microbial opsin ChrimsonR, and showed that the AAV2.7m8 vector had a higher transfection efficiency than AAV2 in retinal ganglion cells (RGCs) and that ChrimsonR fused to tdTomato (ChR-tdT) was expressed more efficiently than ChrimsonR. Light at 600 nm activated RGCs transfected with AAV2.7m8 ChR-tdT, from an irradiance of 10¹⁵ photons.cm⁻².s⁻¹. Vector doses of 5 × 10¹⁰ and 5 × 10¹¹ vg/eye transfected up to 7000 RGCs/mm² in the perifovea, with no significant immune reaction. We recorded RGC responses from a stimulus duration of 1 ms upwards. When using the recorded activity to decode stimulus information, we obtained an estimated visual acuity of 20/249, above the level of legal blindness (20/400). These results lay the groundwork for the ongoing clinical trial with the AAV2.7m8 - ChR-tdT vector for vision restoration in patients with retinitis pigmentosa.

Gauvain et al demonstrate that optogenetic therapy using the AAV2.7m8- ChR-tdT construct can partially restore vision in non-human primates to levels above those considered legally-blind. This study enables the identification of the most suitable construct for ongoing clinical trials attempting vision restoration in patients with retinitis pigmentosa.

Microsaccades in Applied Environments: Real-World Applications of Fixational Eye Movement Measurements

Across a wide variety of research environments, the recording of microsaccades and other fixational eye movements has provided insight and solutions into practical problems. Here we review the literature on fixational eye movements-especially microsaccades-in applied and ecologically-valid scenarios. Recent technical advances allow noninvasive fixational eye movement recordings in real-world contexts, while observers perform a variety of tasks. Thus, fixational eye movement measures have been obtained in a host of real-world scenarios, such as in connection with driver fatigue, vestibular sensory deprivation in astronauts, and elite athletic training, among others. Here we present the state of the art in the practical applications of fixational eye movement research, examine its potential future uses, and discuss the benefits of including microsaccade measures in existing eye movement detection technologies. Current evidence supports the inclusion of fixational eye movement measures in real-world contexts, as part of the development of new or improved oculomotor assessment tools. The real-world applications of fixational eye movement measurements will only grow larger and wider as affordable high-speed and high-spatial resolution eye trackers become increasingly prevalent.

Advanced Circuit and Cellular Imaging Methods in Nonhuman Primates

Novel genetically encoded tools and advanced microscopy methods have revolutionized neural circuit analyses in insects and rodents over the last two decades. Whereas numerous technical hurdles originally barred these methodologies from success in nonhuman primates (NHPs), current research has started to overcome those barriers. In some cases, methodological advances developed with NHPs have even surpassed their precursors. One such advance includes new ultra-large imaging windows on NHP cortex, which are larger than the entire rodent brain and allow analysis unprecedented ultra-large-scale circuits. NHP imaging chambers now remain patent for periods longer than a mouse's lifespan, allowing for long-term all-optical interrogation of identified circuits and neurons over timeframes that are relevant to human cognitive development. Here we present some recent imaging advances brought forth by research teams using macaques and marmosets. These include technical developments in optogenetics; voltage-, calcium- and glutamate-sensitive dye imaging; two-photon and wide-field optical imaging; viral delivery; and genetic expression of indicators and light-activated proteins that result in the visualization of tens of thousands of identified cortical neurons in NHPs. We describe a subset of the many recent advances in circuit and cellular imaging tools in NHPs focusing here primarily on the research presented during the corresponding mini-symposium at the 2019 Society for Neuroscience annual meeting.

Head and Gaze Behavior in Retinitis Pigmentosa

Purpose

Peripheral visual field loss (PVFL) due to retinitis pigmentosa (RP) decreases saccades to areas of visual defect, leading to a habitually confined range of eye movement. We investigated the relative contributions of head and eye movement in RP patients and normal-sighted controls to determine whether this reduced eye movement is offset by increased head movement.

Methods

Eye-head coordination was examined in 18 early-moderate RP patients, 4 late-stage RP patients, and 19 normal-sighted controls. Three metrics were extracted: the extent of eye, head, and total gaze (eye+head) movement while viewing a naturalistic scene; head gain, the ratio of head movement to total gaze movement during smooth pursuit; and the customary oculomotor range (COMR), the orbital range within which the eye is preferentially maintained during a pro-saccade task.

Results

The late-stage RP group had minimal gaze movement and could not discern the naturalistic scene. Variance in head position in early-moderate RP was significantly greater than in controls, whereas variance in total gaze was similar. Head gain was greater in early-moderate RP than in controls, whereas COMR was smaller. Across groups, visual field extent was negatively correlated with head gain and positively correlated with COMR. Accounting for age effects, these results demonstrate increased head movement at the expense of eye movement in participants with PVFL.

Conclusions

RP is associated with an increased propensity for head movement during gaze shifts, and the magnitude of this effect is dependent on the severity of visual field loss.

Contemporary approaches to visual prostheses

Visual prostheses serve to restore visual function following acquired blindness. Acquired blindness (as opposed to congenital blindness) has many causes, including diseases such as retinitis pigmentosa, glaucoma, and macular degeneration, or trauma such as caused by automobile accident or blast damage from explosions. Many of the blindness-causing diseases target the retina or other ocular structure. Often, despite the loss of sensitivity to light, the remainder of the visual pathway is still functional, enabling electrical devices to deliver effective and meaningful visual information to the brain via arrays of electrodes. These arrays can be placed in any part of the early visual pathway, such as the retina, optic nerve, lateral geniculate nucleus, or visual cortex. A camera or other imaging source is used to drive electrical stimulation of remaining healthy cells or structures to create artificial vision and provide restoration of function. In this review, each approach to visual prostheses is described, including advantages and disadvantages as well as assessments of the current state of the art. Most of the work to-date has been targeting stimulation of (a) the retina, with three devices approved for general use and two more in clinical testing; (b) the lateral geniculate nucleus, with efforts still in the pre-clinical stage; and (c) the cortex, with three devices in clinical testing and none currently approved for general use despite the longest history of investigation of the three major approaches. Each class of device has different medical indications, and different levels of invasiveness required for implantation. All contemporary devices deliver relatively poor vision. There has been remarkable progress since the first proof-of-concept demonstration that used stimulation of the primary visual cortex, with the field exploring all viable options for restoration of function. Much of the progress has been recent, driven by advances in microelectronics and biocompatibility. With three devices currently approved for general use in various parts of the world, and a handful of additional devices well along in the pipeline toward approval, prospects for wide deployment of a device-based therapy to treat acquired blindness are good.