A model of ganglion axon pathways accounts for percepts elicited by retinal implants

A virtual patient simulation modeling the neural and perceptual effects of human visual cortical stimulation, from pulse trains to percepts

The field of cortical sight restoration prostheses is making rapid progress with three clinical trials of visual cortical prostheses underway. However, as yet, we have only limited insight into the perceptual experiences produced by these implants. Here we describe a computational model or 'virtual patient', based on the neurophysiological architecture of V1, which successfully predicts the perceptual experience of participants across a wide range of previously published human cortical stimulation studies describing the location, size, brightness and spatiotemporal shape of electrically induced percepts in humans. Our simulations suggest that, in the foreseeable future the perceptual quality of cortical prosthetic devices is likely to be limited by the neurophysiological organization of visual cortex, rather than engineering constraints.

Neural activity shaping in visual prostheses with deep learning

Objective.The visual perception provided by retinal prostheses is limited by the overlapping current spread of adjacent electrodes. This reduces the spatial resolution attainable with unipolar stimulation. Conversely, simultaneous multipolar stimulation guided by the measured neural responses-neural activity shaping (NAS)-can attenuate excessive spread of excitation allowing for more precise control over the pattern of neural activation. However, defining effective multipolar stimulus patterns is a challenging task. Previous attempts focused on analytical solutions based on an assumed linear nonlinear model of retinal response; an analytical model inversion (AMI) approach. Here, we propose a model-free solution for NAS, using artificial neural networks (ANNs) that could be trained with data acquired from the implant.Approach.Our method consists of two ANNs trained sequentially. The measurement predictor network (MPN) is trained on data from the implant and is used to predict how the retina responds to multipolar stimulation. The stimulus generator network is trained on a large dataset of natural images and uses the trained MPN to determine efficient multipolar stimulus patterns by learning its inverse model. We validate our methodin silicousing a realistic model of retinal response to multipolar stimulation.Main results.We show that our ANN-based NAS approach produces sharper retinal activations than the conventional unipolar stimulation strategy. As a theoretical bench-mark of optimal NAS results, we implemented AMI stimulation by inverting the model used to simulate the retina. Our ANN strategy produced equivalent results to AMI, while not being restricted to any specific type of retina model and being three orders of magnitude more computationally efficient.Significance.Our novel protocol provides a method for efficient and personalized retinal stimulation, which may improve the visual experience and quality of life of retinal prosthesis users.

Pre-processing visual scenes for retinal prosthesis systems: A comprehensive review

BACKGROUND

Retinal prostheses offer hope for individuals with degenerative retinal diseases by stimulating the remaining retinal cells to partially restore their vision. This review delves into the current advancements in retinal prosthesis technology, with a special emphasis on the pivotal role that image processing and machine learning techniques play in this evolution.

METHODS

We provide a comprehensive analysis of the existing implantable devices and optogenetic strategies, delineating their advantages, limitations, and challenges in addressing complex visual tasks. The review extends to various image processing algorithms and deep learning architectures that have been implemented to enhance the functionality of retinal prosthetic devices. We also illustrate the testing results by demonstrating the clinical trials or using Simulated Prosthetic Vision (SPV) through phosphene simulations, which is a critical aspect of simulating visual perception for retinal prosthesis users.

RESULTS

Our review highlights the significant progress in retinal prosthesis technology, particularly its capacity to augment visual perception among the visually impaired. It discusses the integration between image processing and deep learning, illustrating their impact on individual interactions and navigations within the environment through applying clinical trials and also illustrating the limitations of some techniques to be used with current devices, as some approaches only use simulation even on sighted-normal individuals or rely on qualitative analysis, where some consider realistic perception models and others do not.

CONCLUSION

This interdisciplinary field holds promise for the future of retinal prostheses, with the potential to significantly enhance the quality of life for individuals with retinal prostheses. Future research directions should pivot towards optimizing phosphene simulations for SPV approaches, considering the distorted and confusing nature of phosphene perception, thereby enriching the visual perception provided by these prosthetic devices. This endeavor will not only improve navigational independence but also facilitate a more immersive interaction with the environment.

Visual Cortical Thickness Increases with Prolonged Artificial Vision Restoration

The Argus II retinal prosthesis restores visual perception to late blind patients. It has been shown that structural changes occur in the brain due to late-onset blindness, including cortical thinning in visual regions of the brain. Following vision restoration, it is not yet known whether these visual regions are reinvigorated and regain a normal cortical thickness or retain the diminished thickness from blindness. We evaluated the cortical thicknesses of ten Argus II Retinal Prostheses patients, ten blind patients, and thirteen sighted participants. The Argus II patients on average had a thicker left Cuneus Cortex and Lateral Occipital Cortex relative to the blind patients. The duration of the Argus II use (time since implant in active users) significantly partially correlated with thicker visual cortical regions in the left hemisphere. Furthermore, in the two case studies (scanned before and after implantation), the patient with longer device use (44.5 months) had an increase in the cortical thickness of visual regions, whereas the shorter-using patient did not (6.5 months). Finally, a third case, scanned at three time points post-implantation, showed an increase in cortical thickness in the Lateral Occipital Cortex between 43.5 and 57 months, which was maintained even after 3 years of disuse (106 months). Overall, the Argus II patients' cortical thickness was on average significantly rejuvenated in two higher visual regions and, patients using the implant for a longer duration had thicker visual regions. This research raises the possibility of structural plasticity reversing visual cortical atrophy in late-blind patients with prolonged vision restoration.

Decoding electroencephalographic responses to visual stimuli compatible with electrical stimulation

Electrical stimulation of the visual nervous system could improve the quality of life of patients affected by acquired blindness by restoring some visual sensations, but requires careful optimization of stimulation parameters to produce useful perceptions. Neural correlates of elicited perceptions could be used for fast automatic optimization, with electroencephalography as a natural choice as it can be acquired non-invasively. Nonetheless, its low signal-to-noise ratio may hinder discrimination of similar visual patterns, preventing its use in the optimization of electrical stimulation. Our work investigates for the first time the discriminability of the electroencephalographic responses to visual stimuli compatible with electrical stimulation, employing a newly acquired dataset whose stimuli encompass the concurrent variation of several features, while neuroscience research tends to study the neural correlates of single visual features. We then performed above-chance single-trial decoding of multiple features of our newly crafted visual stimuli using relatively simple machine learning algorithms. A decoding scheme employing the information from multiple stimulus presentations was implemented, substantially improving our decoding performance, suggesting that such methods should be used systematically in future applications. The significance of the present work relies in the determination of which visual features can be decoded from electroencephalographic responses to electrical stimulation-compatible stimuli and at which granularity they can be discriminated. Our methods pave the way to using electroencephalographic correlates to optimize electrical stimulation parameters, thus increasing the effectiveness of current visual neuroprostheses.

A Cell-Type Selective Stimulation and Recording System for Retinal Ganglion Cells

Future retinal implants will require a stimulation selectivity between different sub-types of Retinal Ganglion Cells (RGCs) to evoke natural perceptions rather than phosphenes in patients. To achieve this, a cell-type specific stimulation pipeline is required that identifies target RGC sub-types from recorded input images and extracts the specific stimulation parameters to activate this cell-type selectively. Promising biological experiments showed that ON-/OFF- sustained/transient RGCs could be selectively activated by modulating repetition rate and amplitude of an electrical stimulation current in the kilohertz range. This research presents a 42 channel current controlled stimulation and recording system on chip (SoC) with parameter input from a real time target RGC selection algorithm. The SoC is able to stimulate retinal tissue with sinusoidal frequencies higher than 1 kHz at amplitudes of up to 200 μA at a supply voltage of 1.8 V. It also includes tunable recording units with an integrated action potential detection pipeline that are able to amplify signals between 1 Hz and 50 kHz. The required area of one stimulator is 0.0071 mm2, while one recording unit consumes an area of 0.0092 mm2. The application of sinusoidal stimulation currents in the kilohertz range towards retinal tissue leads to a suppressive response of only certain RGC sub-types that has not been oberved before, using electrical stimulation. Because this response is very similar to the natural light response of RGCs, this stimulation approach can lead to a more genuine visual perception for patients using retinal implants.

Avoidance of axonal stimulation with sinusoidal epiretinal stimulation

Objective.Neuromodulation, particularly electrical stimulation, necessitates high spatial resolution to achieve artificial vision with high acuity. In epiretinal implants, this is hindered by the undesired activation of distal axons. Here, we investigate focal and axonal activation of retinal ganglion cells (RGCs) in epiretinal configuration for different sinusoidal stimulation frequencies.Approach.RGC responses to epiretinal sinusoidal stimulation at frequencies between 40 and 100 Hz were tested inex-vivophotoreceptor degenerated (rd10) isolated retinae. Experiments were conducted using a high-density CMOS-based microelectrode array, which allows to localize RGC cell bodies and axons at high spatial resolution.Main results.We report current and charge density thresholds for focal and distal axon activation at stimulation frequencies of 40, 60, 80, and 100 Hz for an electrode size with an effective area of 0.01 mm2. Activation of distal axons is avoided up to a stimulation amplitude of 0.23µA (corresponding to 17.3µC cm-2) at 40 Hz and up to a stimulation amplitude of 0.28µA (14.8µC cm-2) at 60 Hz. The threshold ratio between focal and axonal activation increases from 1.1 for 100 Hz up to 1.6 for 60 Hz, while at 40 Hz stimulation frequency, almost no axonal responses were detected in the tested intensity range. With the use of synaptic blockers, we demonstrate the underlying direct activation mechanism of the ganglion cells. Finally, using high-resolution electrical imaging and label-free electrophysiological axon tracking, we demonstrate the extent of activation in axon bundles.Significance.Our results can be exploited to define a spatially selective stimulation strategy avoiding axonal activation in future retinal implants, thereby solving one of the major limitations of artificial vision. The results may be extended to other fields of neuroprosthetics to achieve selective focal electrical stimulation.

Axonal stimulation affects the linear summation of single-point perception in three Argus II users

Objective.Retinal implants use electrical stimulation to elicit perceived flashes of light ('phosphenes'). Single-electrode phosphene shape has been shown to vary systematically with stimulus parameters and the retinal location of the stimulating electrode, due to incidental activation of passing nerve fiber bundles. However, this knowledge has yet to be extended to paired-electrode stimulation.Approach.We retrospectively analyzed 3548 phosphene drawings made by three blind participants implanted with an Argus II Retinal Prosthesis. Phosphene shape (characterized by area, perimeter, major and minor axis length) and number of perceived phosphenes were averaged across trials and correlated with the corresponding single-electrode parameters. In addition, the number of phosphenes was correlated with stimulus amplitude and neuroanatomical parameters: electrode-retina and electrode-fovea distance as well as the electrode-electrode distance to ('between-axon') and along axon bundles ('along-axon'). Statistical analyses were conducted using linear regression and partial correlation analysis.Main results.Simple regression revealed that each paired-electrode shape descriptor could be predicted by the sum of the two corresponding single-electrode shape descriptors (p < .001). Multiple regression revealed that paired-electrode phosphene shape was primarily predicted by stimulus amplitude and electrode-fovea distance (p < .05). Interestingly, the number of elicited phosphenes tended to increase with between-axon distance (p < .05), but not with along-axon distance, in two out of three participants.Significance.The shape of phosphenes elicited by paired-electrode stimulation was well predicted by the shape of their corresponding single-electrode phosphenes, suggesting that two-point perception can be expressed as the linear summation of single-point perception. The impact of the between-axon distance on the perceived number of phosphenes provides further evidence in support of the axon map model for epiretinal stimulation. These findings contribute to the growing literature on phosphene perception and have important implications for the design of future retinal prostheses.

Explainable machine learning predictions of perceptual sensitivity for retinal prostheses

Objective.Retinal prostheses evoke visual precepts by electrically stimulating functioning cells in the retina. Despite high variance in perceptual thresholds across subjects, among electrodes within a subject, and over time, retinal prosthesis users must undergo 'system fitting', a process performed to calibrate stimulation parameters according to the subject's perceptual thresholds. Although previous work has identified electrode-retina distance and impedance as key factors affecting thresholds, an accurate predictive model is still lacking.Approach.To address these challenges, we (1) fitted machine learning models to a large longitudinal dataset with the goal of predicting individual electrode thresholds and deactivation as a function of stimulus, electrode, and clinical parameters ('predictors') and (2) leveraged explainable artificial intelligence (XAI) to reveal which of these predictors were most important.Main results.Our models accounted for up to 76% of the perceptual threshold response variance and enabled predictions of whether an electrode was deactivated in a given trial with F1 and area under the ROC curve scores of up to 0.732 and 0.911, respectively. Our models identified novel predictors of perceptual sensitivity, including subject age, time since blindness onset, and electrode-fovea distance.Significance.Our results demonstrate that routinely collected clinical measures and a single session of system fitting might be sufficient to inform an XAI-based threshold prediction strategy, which has the potential to transform clinical practice in predicting visual outcomes.

Fabrication and Validation of Sub-Cellular Carbon Fiber Electrodes

Multielectrode arrays for interfacing with neurons are of great interest for a wide range of medical applications. However, current electrodes cause damage over time. Ultra small carbon fibers help to address issues but controlling the electrode site geometry is difficult. Here we propose a methodology to create small, pointed fiber electrodes (SPFe). We compare the SPFe to previously made blowtorched fibers in characterization. The SPFe result in small site sizes [Formula: see text] with consistently sharp points (20.8 ± 7.64°). Additionally, these electrodes were able to record and/or stimulate neurons multiple animal models including rat cortex, mouse retina, Aplysia ganglia and octopus axial cord. In rat cortex, these electrodes recorded significantly higher peak amplitudes than the traditional blowtorched fibers. These SPFe may be applicable to a wide range of applications requiring a highly specific interface with individual neurons.

Learning to see via epiretinal implant stimulationin silicowith model-based deep reinforcement learning

OBJECTIVE

Diseases such as age-related macular degeneration and retinitis pigmentosa cause the degradation of the photoreceptor layer. One approach to restore vision is to electrically stimulate the surviving retinal ganglion cells with a microelectrode array such as epiretinal implants. Epiretinal implants are known to generate visible anisotropic shapes elongated along the axon fascicles of neighboring retinal ganglion cells. Recent work has demonstrated that to obtain isotropic pixel-like shapes, it is possible to map axon fascicles and avoid stimulating them by inactivating electrodes or lowering stimulation current levels. Avoiding axon fascicule stimulation aims to remove brushstroke-like shapes in favor of a more reduced set of pixel-like shapes.

APPROACH

In this study, we propose the use of isotropic and anisotropic shapes to render intelligible images on the retina of a virtual patient in a reinforcement learning environment named rlretina. The environment formalizes the task as using brushstrokes in a stroke-based rendering task.

MAIN RESULTS

We train a deep reinforcement learning agent that learns to assemble isotropic and anisotropic shapes to form an image. We investigate which error-based or perception-based metrics are adequate to reward the agent. The agent is trained in a model-based data generation fashion using the psychophysically validated axon map model to render images as perceived by different virtual patients. We show that the agent can generate more intelligible images compared to the naive method in different virtual patients.

SIGNIFICANCE

This work shares a new way to address epiretinal stimulation that constitutes a first step towards improving visual acuity in artificially-restored vision using anisotropic phosphenes.

Patient-specific computational models of retinal prostheses

Retinal prostheses stimulate inner retinal neurons to create visual perception for blind patients. Implanted arrays have many small electrodes. Not all electrodes induce perception at the same stimulus amplitude, requiring clinicians to manually establish a visual perception threshold for each one. Phosphenes created by single-electrode stimuli can also vary in shape, size, and brightness. Computational models provide a tool to predict inter-electrode variability and automate device programming. In this study, we created statistical and patient-specific field-cable models to investigate inter-electrode variability across seven epiretinal prosthesis users. Our statistical analysis revealed that retinal thickness beneath the electrode correlated with perceptual threshold, with a significant fixed effect across participants. Electrode-retina distance and electrode impedance also correlated with perceptual threshold for some participants, but these effects varied by individual. We developed a novel method to construct patient-specific field-cable models from optical coherence tomography images. Predictions with these models significantly correlated with perceptual threshold for 80% of participants. Additionally, we demonstrated that patient-specific field-cable models could predict retinal activity and phosphene size. These computational models could be beneficial for determining optimal stimulation settings in silico, circumventing the trial-and-error testing of a large parameter space in clinic.

Optogenetic targeting of AII amacrine cells restores retinal computations performed by the inner retina

Most inherited retinal dystrophies display progressive photoreceptor cell degeneration leading to severe visual impairment. Optogenetic reactivation of inner retinal neurons is a promising avenue to restore vision in retinas having lost their photoreceptors. Expression of optogenetic proteins in surviving ganglion cells, the retinal output, allows them to take on the lost photoreceptive function. Nonetheless, this creates an exclusively ON retina by expression of depolarizing optogenetic proteins in all classes of ganglion cells, whereas a normal retina extracts several features from the visual scene, with different ganglion cells detecting light increase (ON) and light decrease (OFF). Refinement of this therapeutic strategy should thus aim at restoring these computations. Here we used a vector that targets gene expression to a specific interneuron of the retina called the AII amacrine cell. AII amacrine cells simultaneously activate the ON pathway and inhibit the OFF pathway. We show that the optogenetic stimulation of AII amacrine cells allows restoration of both ON and OFF responses in the retina, but also mediates other types of retinal processing such as sustained and transient responses. Targeting amacrine cells with optogenetics is thus a promising avenue to restore better retinal function and visual perception in patients suffering from retinal degeneration.

Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses

Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to optimize patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. We show that our approach quickly learns a personalized stimulus encoder, leads to dramatic improvements in the quality of restored vision, and is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.

Suppressing Retinal Remodeling to Mitigate Vision Loss in Photoreceptor Degenerative Disorders

Rod and cone photoreceptors degenerate in retinitis pigmentosa and age-related macular degeneration, robbing the visual system of light-triggered signals necessary for sight. However, changes in the retina do not stop with the photoreceptors. A stereotypical set of morphological and physiological changes, known as remodeling, occur in downstream retinal neurons. Some aspects of remodeling are homeostatic, with structural or functional changes compensating for partial loss of visual inputs. However, other aspects are nonhomeostatic, corrupting retinal information processing to obscure vision mediated naturally by surviving photoreceptors or artificially by vision-restoration technologies. In this review, I consider the mechanism of remodeling and its consequences for residual and restored visual function; discuss the role of retinoic acid, a critical molecular trigger of detrimental remodeling; and discuss strategies for suppressing retinoic acid biosynthesis or signaling as therapeutic possibilities for mitigating vision loss.

Patient-specific computational models of retinal prostheses

Retinal prostheses stimulate inner retinal neurons to create visual perception for blind patients. Implanted arrays have many small electrodes, which act as pixels. Not all electrodes induce perception at the same stimulus amplitude, requiring clinicians to manually establish a visual perception threshold for each one. Phosphenes created by single-electrode stimuli can also vary in shape, size, and brightness. Computational models provide a tool to predict inter-electrode variability and automate device programming. In this study, we created statistical and patient-specific field-cable models to investigate inter-electrode variability across seven epiretinal prosthesis users. Our statistical analysis revealed that retinal thickness beneath the electrode correlated with perceptual threshold, with a significant fixed effect across participants. Electrode-retina distance and electrode impedance also correlated with perceptual threshold for some participants, but these effects varied by individual. We developed a novel method to construct patient-specific field-cable models from optical coherence tomography images. Predictions with these models significantly correlated with perceptual threshold for 80% of participants. Additionally, we demonstrated that patient-specific field-cable models could predict retinal activity and phosphene size. These computational models could be beneficial for determining optimal stimulation settings in silico, circumventing the trial-and-error testing of a large parameter space in clinic.

A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.

Retinal ganglion cells undergo cell type-specific functional changes in a computational model of cone-mediated retinal degeneration

Introduction

Understanding the retina in health and disease is a key issue for neuroscience and neuroengineering applications such as retinal prostheses. During degeneration, the retinal network undergoes complex and multi-stage neuroanatomical alterations, which drastically impact the retinal ganglion cell (RGC) response and are of clinical importance. Here we present a biophysically detailed in silico model of the cone pathway in the retina that simulates the network-level response to both light and electrical stimulation.

Methods

The model included 11, 138 cells belonging to nine different cell types (cone photoreceptors, horizontal cells, ON/OFF bipolar cells, ON/OFF amacrine cells, and ON/OFF ganglion cells) confined to a 300 × 300 × 210μm patch of the parafoveal retina. After verifying that the model reproduced seminal findings about the light response of retinal ganglion cells (RGCs), we systematically introduced anatomical and neurophysiological changes (e.g., reduced light sensitivity of photoreceptor, cell death, cell migration) to the network and studied their effect on network activity.

Results

The model was not only able to reproduce common findings about RGC activity in the degenerated retina, such as hyperactivity and increased electrical thresholds, but also offers testable predictions about the underlying neuroanatomical mechanisms.

Discussion

Overall, our findings demonstrate how biophysical changes typified by cone-mediated retinal degeneration may impact retinal responses to light and electrical stimulation. These insights may further our understanding of retinal processing and inform the design of retinal prostheses.

A systematic review of extended reality (XR) for understanding and augmenting vision loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user's eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

Pulse trains to percepts: A virtual patient describing the perceptual effects of human visual cortical stimulation

The field of cortical sight restoration prostheses is making rapid progress with three clinical trials of visual cortical prostheses underway. However, as yet, we have only limited insight into the perceptual experiences produced by these implants. Here we describe a computational model or 'virtual patient', based on the neurophysiological architecture of V1, which successfully predicts the perceptual experience of participants across a wide range of previously published cortical stimulation studies describing the location, size, brightness and spatiotemporal shape of electrically induced percepts in humans. Our simulations suggest that, in the foreseeable future the perceptual quality of cortical prosthetic devices is likely to be limited by the neurophysiological organization of visual cortex, rather than engineering constraints.

Estimating Phosphene Locations Using Eye Movements of Suprachoroidal Retinal Prosthesis Users

Purpose

Accurate mapping of phosphene locations from visual prostheses is vital to encode spatial information. This process may involve the subject pointing to evoked phosphene locations with their finger. Here, we demonstrate phosphene mapping for a retinal implant using eye movements and compare it with retinotopic electrode positions and previous results using conventional finger-based mapping.

Methods

Three suprachoroidal retinal implant recipients (NCT03406416) indicated the spatial position of phosphenes. Electrodes were stimulated individually, and the subjects moved their finger (finger based) or their eyes (gaze based) to the perceived phosphene location. The distortion of the measured phosphene locations from the expected locations (retinotopic electrode locations) was characterized with Procrustes analysis.

Results

The finger-based phosphene locations were compressed spatially relative to the expected locations all three subjects, but preserved the general retinotopic arrangement (scale factors ranged from 0.37 to 0.83). In two subjects, the gaze-based phosphene locations were similar to the expected locations (scale factors of 0.72 and 0.99). For the third subject, there was no apparent relationship between gaze-based phosphene locations and electrode locations (scale factor of 0.07).

Conclusions

Gaze-based phosphene mapping was achievable in two of three tested retinal prosthesis subjects and their derived phosphene maps correlated well with the retinotopic electrode layout. A third subject could not produce a coherent gaze-based phosphene map, but this may have revealed that their phosphenes were indistinct spatially.

Translational Relevance

Gaze-based phosphene mapping is a viable alternative to conventional finger-based mapping, but may not be suitable for all subjects.

Long term positional stability of the Argus II retinal prosthesis epiretinal implant

BACKGROUND

The Argus II Retinal Prosthesis System (Second Sight Medical Products, Sylmar, California) is an epiretinal prosthesis that serves to provide useful vision to people who are affected by retinal degenerative diseases such as retinitis pigmentosa (RP). The purpose of this study was to analyze postoperative movement of the electrode array.

METHODS

Five patients diagnosed with profound retinal dystrophy who have undergone implantation of retinal prosthesis at Stony Brook University Hospital. Fundoscopy was performed at postoperative month 1 (M1), month 3 (M3), month 6 (M6), month 12 (M12), and month 24 (M24) visits. Fundoscopy was extracted and analyzed via NIH ImageJ. Data analysis was completed using IBM SPSS. Various lengths and angles were measured each postoperative month using ImageJ.

RESULTS

There was no significant change in distance between the optic disc and the surgical handle (length AB) over the two-year span (F = 0.196, p = 0.705). There was a significant change in distance of length AB over time between patients between M3 and M6 (p = 0.025). A repeated measures ANOVA revealed that there was statistically significant change of the optic disc-tack-surgical handle angle (𝛾) (M1 to M24) (F = 3.527, p = 0.030). There was no significant change in angle 𝜟 (the angle to the horizontal of the image), angle 𝜶 (tack-optic disc-surgical handle), and angle 𝜷 (optic-disc-surgical handle-tack).

CONCLUSION

Our results demonstrate that there may be postoperative movement of the retinal prosthesis over time, as a statistically significant downward rotation is reported over the 2 years span. It is important, moving forward, to further study this movement and to take into consideration such movement when designing retinal implants. It is important to note that this study is limited by the small sample size, and therefore, the conclusions drawn are limited.

Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction

Objective. Retinal prostheses are promising devices to restore vision for patients with severe age-related macular degeneration or retinitis pigmentosa disease. The visual processing mechanism embodied in retinal prostheses play an important role in the restoration effect. Its performance depends on our understanding of the retina's working mechanism and the evolvement of computer vision models. Recently, remarkable progress has been made in the field of processing algorithm for retinal prostheses where the new discovery of the retina's working principle and state-of-the-arts computer vision models are combined together.Approach. We investigated the related research on artificial intelligence techniques for retinal prostheses. The processing algorithm in these studies could be attributed to three types: computer vision-related methods, biophysical models, and deep learning models.Main results. In this review, we first illustrate the structure and function of the normal and degenerated retina, then demonstrate the vision rehabilitation mechanism of three representative retinal prostheses. It is necessary to summarize the computational frameworks abstracted from the normal retina. In addition, the development and feature of three types of different processing algorithms are summarized. Finally, we analyze the bottleneck in existing algorithms and propose our prospect about the future directions to improve the restoration effect.Significance. This review systematically summarizes existing processing models for predicting the response of the retina to external stimuli. What's more, the suggestions for future direction may inspire researchers in this field to design better algorithms for retinal prostheses.

Explainable Machine Learning Predictions of Perceptual Sensitivity for Retinal Prostheses

To provide appropriate levels of stimulation, retinal prostheses must be calibrated to an individual's perceptual thresholds ('system fitting'), despite thresholds varying drastically across subjects, across electrodes within a subject, and over time. Although previous work has identified electrode-retina distance and impedance as key factors affecting thresholds, an accurate predictive model is still lacking. To address these challenges, we 1) fitted machine learning (ML) models to a large longitudinal dataset with the goal of predicting individual electrode thresholds and deactivation as a function of stimulus, electrode, and clinical parameters ('predictors') and 2) leveraged explainable artificial intelligence (XAI) to reveal which of these predictors were most important. Our models accounted for up to 77% of the perceptual threshold response variance and enabled predictions of whether an electrode was deactivated in a given trial with F1 and AUC scores of up to 0.740 and 0.913, respectively. Deactivation and threshold models identified novel predictors of perceptual sensitivity, including subject age, time since blindness onset, and electrode-fovea distance. Our results demonstrate that routinely collected clinical measures and a single session of system fitting might be sufficient to inform an XAI-based threshold prediction strategy, which may transform clinical practice in predicting visual outcomes.

High spatial resolution artificial vision inferred from the spiking output of retinal ganglion cells stimulated by optogenetic and electrical means

With vision impairment affecting millions of people world-wide, various strategies aiming at vision restoration are being undertaken. Thanks to decades of extensive research, electrical stimulation approaches to vision restoration began to undergo clinical trials. Quite recently, another technique employing optogenetic therapy emerged as a possible alternative. Both artificial vision restoration strategies reported poor spatial resolution so far. In this article, we compared the spatial resolution inferred ex vivo under ideal conditions using a computational model analysis of the retinal ganglion cell (RGC) spiking activity. The RGC spiking was stimulated in epiretinal configuration by either optogenetic or electrical means. RGCs activity was recorded from the ex vivo retina of transgenic late-stage photoreceptor-degenerated mice (rd10) using a high-density Complementary Metal Oxide Semiconductor (CMOS) based microelectrode array. The majority of retinal samples were stimulated by both, optogenetic and electrical stimuli using a spatial grating stimulus. A population-level analysis of the spiking activity of identified RGCs was performed and the spatial resolution achieved through electrical and optogenetic photo-stimulation was inferred using a support vector machine classifier. The best f1 score of the classifier for the electrical stimulation in epiretinal configuration was 86% for 32 micron wide gratings and increased to 100% for 128 microns. For optogenetically activated cells, we obtained high f1 scores of 82% for 10 microns grid width for a photo-stimulation frequency of 2.5 Hz and 73% for a photo-stimulation frequency of 10 Hz. A subsequent analysis, considering only the RGCs modulated in both electrical and optogenetic stimulation protocols revealed no significant difference in the prediction accuracy between the two stimulation modalities. The results presented here indicate that a high spatial resolution can be achieved for electrical or optogenetic artificial stimulation using the activated retinal ganglion cell output.

Towards aSmart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Objective.How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies.Approach.Rather than aiming to represent the visual scene as naturally as possible, aSmart Bionic Eyecould provide visual augmentations through the means of artificial intelligence-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care.Main results.Complementary to existing research aiming to restore natural vision, we propose a patient-centered approach to incorporate deep learning-based visual augmentations into the next generation of devices.Significance.The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.

Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capabilities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here, we frame this as an end-to-end optimization problem, where a deep neural network stimulus encoder is trained to invert a known and fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of this hybrid neural autoencoder (HNA) in visual neuroprostheses. We find that HNA produces high-fidelity patient-specific stimuli representing handwritten digits and segmented images of everyday objects, and significantly outperforms conventional encoding strategies across all simulated patients. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.

Short pulses of epiretinal prostheses evoke network-mediated responses in retinal ganglion cells by stimulating presynaptic neurons

OBJECTIVE

Microelectronic retinal implant aims to restore functional vision with electric stimulation. Short pulses are generally known to directly activate retinal ganglion cells (RGCs) with a notion of one or two spike(s) per pulse. In the present work, we systematically explore network-mediated responses that arise from various short pulses in both normal and degenerate retinas.

APPROACH

Cell-attached patch clamping was used to record spiking responses of RGCs in wild-type (C57BL/6J) and retinal degeneration (rd10) mice. Alpha RGCs of the mouse retinas were targeted by their large soma sizes and classified by their responses to spot flashes. Then, RGCs were electrically stimulated by various conditions such as duration (100-460 μs), count (1-10), amplitude (100-400 μA), and repeating frequency (10-40 Hz) of short pulses. Also, their responses were compared with each own response to a single 4-ms-long pulse which is known to evoke strong indirect responses.

MAIN RESULTS

Short pulses evoked strong network-mediated responses not only in both ON and OFF types of RGCs in the healthy retinas but also in RGCs of the severely degenerate retina. However, the spike timing consistency across repeats not decreased significantly in the rd10 RGCs compared to the healthy ON and OFF RGCs. Network-mediated responses of ON RGCs were highly dependent on the current amplitude of stimuli but much less on the pulse count and the repetition frequency. In contrast, responses of OFF RGCs were more influenced by the number of stimuli than the current amplitude.

SIGNIFICANCE

Our results demonstrate that short pulses also elicit indirect responses by activating presynaptic neurons. In the case of the commercial retinal prostheses using repeating short pulses, there is a possibility that the performance of clinical devices is highly related to the preserved retinal circuits. Therefore, examination of surviving retinal neurons in patients would be necessary to improve the efficacy of retinal prostheses.

Clinical Progress and Optimization of Information Processing in Artificial Visual Prostheses

Visual prostheses, used to assist in restoring functional vision to the visually impaired, convert captured external images into corresponding electrical stimulation patterns that are stimulated by implanted microelectrodes to induce phosphenes and eventually visual perception. Detecting and providing useful visual information to the prosthesis wearer under limited artificial vision has been an important concern in the field of visual prosthesis. Along with the development of prosthetic device design and stimulus encoding methods, researchers have explored the possibility of the application of computer vision by simulating visual perception under prosthetic vision. Effective image processing in computer vision is performed to optimize artificial visual information and improve the ability to restore various important visual functions in implant recipients, allowing them to better achieve their daily demands. This paper first reviews the recent clinical implantation of different types of visual prostheses, summarizes the artificial visual perception of implant recipients, and especially focuses on its irregularities, such as dropout and distorted phosphenes. Then, the important aspects of computer vision in the optimization of visual information processing are reviewed, and the possibilities and shortcomings of these solutions are discussed. Ultimately, the development direction and emphasis issues for improving the performance of visual prosthesis devices are summarized.

Factors affecting two-point discrimination in Argus II patients

Two of the main obstacles to the development of epiretinal prosthesis technology are electrodes that require current amplitudes above safety limits to reliably elicit percepts, and a failure to consistently elicit pattern vision. Here, we explored the causes of high current amplitude thresholds and poor spatial resolution within the Argus II epiretinal implant. We measured current amplitude thresholds and two-point discrimination (the ability to determine whether one or two electrodes had been stimulated) in 3 blind participants implanted with Argus II devices. Our data and simulations show that axonal stimulation, lift and retinal damage all play a role in reducing performance in the Argus 2, by either limiting sensitivity and/or reducing spatial resolution. Understanding the relative role of these various factors will be critical for developing and surgically implanting devices that can successfully subserve pattern vision.

Preferential modulation of individual retinal ganglion cells by electrical stimulation

OBJECTIVE

Retinal prostheses have been able to recover partial vision in blind patients with retinal degeneration by electrically stimulating surviving cells in the retina, such as retinal ganglion cells (RGCs), but the restored vision is limited. This is partly due to non-preferential stimulation of all RGCs near a single stimulating electrode, which include cells that conflict in their response properties and their contribution to the vision process. Our study proposes a stimulation strategy to preferentially stimulate individual RGCs based on their temporal electrical receptive fields (tERFs).

APPROACH

We recorded the responses of RGCs using whole-cell current-clamp and demonstrated the stimulation strategy, first using intracellular stimulation, then via extracellular stimulation.

MAIN RESULTS

We successfully reconstructed the tERFs according to the RGC response to Gaussian white noise current stimulation. The characteristics of the tERFs were extracted and compared according to the morphological and light response types of the cells. By re-delivering stimulation trains that are composed of the tERFs obtained from different cells, we could target individual RGCs as the cells showed lower activation thresholds to their own tERFs.

SIGNIFICANCE

This proposed stimulation strategy implemented in the next generation of recording and stimulating retinal prostheses may improve the quality of artificial vision.

Human-in-the-loop optimization of visual prosthetic stimulation

Retinal prostheses are a promising strategy to restore sight to patients with retinal degenerative diseases. These devices compensate for the loss of photoreceptors by electrically stimulating neurons in the retina. Currently, the visual function that can be recovered with such devices is very limited. This is due, in part, to current spread, unintended axonal activation, and the limited resolution of existing devices. Here we show, using a recent model of prosthetic vision, that optimizing how visual stimuli are encoded by the device can help overcome some of these limitations, leading to dramatic improvements in visual perception.

APPROACH

We propose a strategy to do this in practice, using patients' feedback in a visual task. The main challenge of our approach comes from the fact that, typically, one only has access to a limited number of noisy responses from patients. We propose two ways to deal with this: first, we use a model of prosthetic vision to constrain and simplify the optimization. We show that, if one knew the parameters of this model for a given patient, it would be possible to greatly improve their perceptual performance. Second we propose a preferential Bayesian optimization to efficiently learn these model parameters for each patient, using minimal trials.

MAIN RESULTS

To test our approach, we presented healthy subjects with visual stimuli generated by a recent model of prosthetic vision, to replicate the perceptual experience of patients fitted with an implant. Our optimization procedure led to significant and robust improvements in perceived image quality, that transferred to increased performance in other tasks.

SIGNIFICANCE

Importantly, our strategy is agnostic to the type of prosthesis and thus could readily be implemented in existing implants.

Sequential epiretinal stimulation improves discrimination in simple shape discrimination tasks only

OBJECTIVE

Electrical stimulation of the retina can elicit flashes of light called phosphenes, which can be used to restore rudimentary vision for people with blindness. Functional sight requires stimulation of multiple electrodes to create patterned vision, but phosphenes tend to merge together in an uninterpretable way. Sequentially stimulating electrodes in human visual cortex has recently demonstrated that shapes could be "drawn" with better perceptual resolution relative to simultaneous stimulation. The goal of this study was to evaluate if sequential stimulation would also form clearer shapes when the retina is the neural target.

APPROACH

Two human participants with retinitis pigmentosa who had Argus® II retinal prostheses participated in this study. We evaluated different temporal parameters for sequential stimulation in phosphene shape mapping and forced-choice discrimination tasks. For the discrimination tasks, performance was compared between stimulating electrodes simultaneously versus sequentially.

MAIN RESULTS

Phosphenes elicited by different electrodes were reported as vastly different shapes. Sequential electrode stimulation outperformed simultaneous stimulation in simple discrimination tasks, in which shapes were created by stimulating 3-4 electrodes, but not in more complex discrimination tasks involving 5+ electrodes. For sequential stimulation, the optimal pulse train duration was 200 ms when stimulating at 20 Hz and the optimal gap interval was tied between 0 and 50 ms. Efficacy of sequential stimulation also depended strongly on selecting electrodes that elicited phosphenes with similar shapes and sizes.

SIGNIFICANCE

An epiretinal prosthesis can produce coherent simple shapes with a sequential stimulation paradigm, which can be used as rudimentary visual feedback. However, success in creating more complex shapes, such as letters of the alphabet, is still limited. Sequential stimulation may be most beneficial for epiretinal prostheses in simple tasks, such as basic navigation, rather than complex tasks such as object identification.

Improving the spatial resolution of artificial vision using midget retinal ganglion cell populations modelled at the human fovea

OBJECTIVE

Retinal prostheses seek to create artificial vision by stimulating surviving retinal neurons of patients with profound vision impairment. Notwithstanding tremendous research efforts, the performance of all implants tested to date has remained rudimentary, incapable of overcoming the threshold for legal blindness. To maximize the perceptual efficacy of retinal prostheses, a device must be capable of controlling retinal neurons with greater spatiotemporal precision. Most studies of retinal stimulation were derived from either non-primate species or the peripheral primate retina. We investigated if artificial stimulation could leverage the high spatial resolution afforded by the neural substrates at the primate fovea and surrounding regions to achieve improved percept qualities.

APPROACH

We began by developing a new computational model capable of generating anatomically accurate retinal ganglion cell (RGC) populations within the human central retina. Next, multiple RGC populations across the central retina were stimulated in-silico to compare clinical and recently proposed neurostimulation configurations based on their ability to improve perceptual efficacy and reduce activation thresholds.

MAIN RESULTS

Our model uniquely upholds eccentricity-dependent characteristics such as RGC density and dendritic field diameter, whilst incorporating anatomically accurate features such as axon projection and three-dimensional RGC layering, features often forgone in favor of reduced computational complexity. Following epiretinal stimulation, the RGCs in our model produced response patterns in shapes akin to the complex percepts reported in clinical trials. Our results also demonstrated that even within the neuron-dense central retina, epiretinal stimulation using a multi-return hexapolar electrode arrangement could reliably achieve spatially focused RGC activation and could achieve single-cell excitation in 74% of all tested locations.

SIGNIFICANCE

This study establishes an anatomically accurate three-dimensional model of the human central retina and demonstrates the potential for an epiretinal hexapolar configuration to achieve consistent, spatially confined retinal responses, even within the neuron-dense foveal region. Our results promote the prospect and optimization of higher spatial resolution in future epiretinal implants.

Simulating the perceptual effects of electrode-retina distance in prosthetic vision

OBJECTIVE

Retinal prostheses aim to restore some vision in retinitis pigmentosa and age-related macular degeneration blind patients. Many spatial and temporal aspects have been found to affect prosthetic vision. Our objective is to study the impact of the space-variant distance between the stimulating electrodes and the surface of the retina on prosthetic vision and how to mitigate this impact.

APPROACH

A prosthetic vision simulation was built to demonstrate the perceptual effects of the electrode-retina distance (ERD) with different random spatial variations, such as size, brightness, shape, dropout, and spatial shifts. Three approaches for reducing the ERD effects are demonstrated: electrode grouping (quads), ERD-based input-image enhancement, and object scanning with and without phosphene persistence. A quantitative assessment for the first two approaches was done based on experiments with 20 subjects and three vision-based computational image similarity metrics.

MAIN RESULTS

The effects of various ERDs on phosphenes' size, brightness, and shape were simulated. Quads, chosen according to the ERDs, effectively elicit phosphenes without exceeding the safe charge density limit, whereas single electrodes with large ERD cannot do so. Input-image enhancement reduced the ERD effects effectively. These two approaches significantly improved ERD-affected prosthetic vision according to the experiment and image similarity metrics. A further reduction of the ERD effects was achieved by scanning an object while moving the head.

SIGNIFICANCE

ERD has multiple effects on perception with retinal prostheses. One of them is vision loss caused by the incapability of electrodes with large ERD to evoke phosphenes. The three approaches presented in this study can be used separately or together to mitigate the impact of ERD. A consideration of our approaches in reducing the perceptual effects of the ERD may help improve the perception with current prosthetic technology and influence the design of future prostheses.

Correlated Activity in the Degenerate Retina Inhibits Focal Response to Electrical Stimulation

Retinal prostheses have shown some clinical success in patients with retinitis pigmentosa and age-related macular degeneration. However, even after the implantation of a retinal prosthesis, the patient’s visual acuity is at best less than 20/420. Reduced visual acuity may be explained by a decrease in the signal-to-noise ratio due to the spontaneous hyperactivity of retinal ganglion cells (RGCs) found in degenerate retinas. Unfortunately, abnormal retinal rewiring, commonly observed in degenerate retinas, has rarely been considered for the development of retinal prostheses. The purpose of this study was to investigate the aberrant retinal network response to electrical stimulation in terms of the spatial distribution of the electrically evoked RGC population. An 8 × 8 multielectrode array was used to measure the spiking activity of the RGC population. RGC spikes were recorded in wild-type [C57BL/6J; P56 (postnatal day 56)], rd1 (P56), rd10 (P14 and P56) mice, and macaque [wild-type and drug-induced retinal degeneration (RD) model] retinas. First, we performed a spike correlation analysis between RGCs to determine RGC connectivity. No correlation was observed between RGCs in the control group, including wild-type mice, rd10 P14 mice, and wild-type macaque retinas. In contrast, for the RD group, including rd1, rd10 P56, and RD macaque retinas, RGCs, up to approximately 400–600 μm apart, were significantly correlated. Moreover, to investigate the RGC population response to electrical stimulation, the number of electrically evoked RGC spikes was measured as a function of the distance between the stimulation and recording electrodes. With an increase in the interelectrode distance, the number of electrically evoked RGC spikes decreased exponentially in the control group. In contrast, electrically evoked RGC spikes were observed throughout the retina in the RD group, regardless of the inter-electrode distance. Taken together, in the degenerate retina, a more strongly coupled retinal network resulted in the widespread distribution of electrically evoked RGC spikes. This finding could explain the low-resolution vision in prosthesis-implanted patients.

Morphological Factors that Underlie Neural Sensitivity to Stimulation in the Retina

Retinal prostheses are a promising therapeutic intervention for patients afflicted by outer retinal degenerative diseases like retinitis pigmentosa and age-related macular degeneration. While significant advances in the development of retinal implants have been made, the quality of vision elicited by these devices remains largely sub-optimal. The variability in the responses produced by retinal devices is most likely due to the differences between the natural cell type-specific signaling that occur in the healthy retina vs. the non-specific activation of multiple cell types arising from artificial stimulation. In order to replicate these natural signaling patterns, stimulation strategies must be capable of preferentially activating specific RGC types. To design more selective stimulation strategies, a better understanding of the morphological factors that underlie the sensitivity to prosthetic stimulation must be developed. This review will focus on the role that different anatomical components play in driving the direct activation of RGCs by extracellular stimulation. Briefly, it will (1) characterize the variability in morphological properties of α-RGCs, (2) detail the influence of morphology on the direct activation of RGCs by electric stimulation, and (3) describe some of the potential biophysical mechanisms that could explain differences in activation thresholds and electrically evoked responses between RGC types.

An in-silico analysis of electrically-evoked responses of midget and parasol retinal ganglion cells in different retinal regions

BACKGROUND

Visual outcomes provided by present retinal prostheses that primarily target retinal ganglion cells (RGCs) through epiretinal stimulation remain rudimentary, partly due to the limited knowledge of retinal responses under electrical stimulation. Better understanding of how different retinal regions can be quantitatively controlled with high spatial accuracy, will be beneficial to the design of micro-electrode arrays (MEAs) and stimulation strategies for next-generation wide-view, high-resolution epiretinal implants.

METHODS

A computational model was developed to assess neural activity at different eccentricities (2 mm and 5 mm) within the human retina. This model included midget and parasol RGCs with anatomically accurate cell distribution and cell-specific morphological information. We then performed in silico investigations of region-specific RGC responses to epiretinal electrical stimulation using varied electrode sizes (5 µm - 210 µm diameter), emulating both commercialized retinal implants and recently-developed prototype devices.

RESULTS

Our model of epiretinal stimulation predicted RGC population excitation analogous to the complex percepts reported in human subjects. Following this, our simulations suggest that midget and parasol RGCs have characteristic regional differences in excitation under preferred electrode sizes. Relatively central (2 mm) regions demonstrated higher number of excited RGCs but lower overall activated receptive field (RF) areas under the same stimulus amplitudes (two-way ANOVA, p < 0.05). Furthermore, the activated RGC numbers per unit active RF area (number-RF ratio) were significantly higher in central than in peripheral regions, and higher in the midget than in the parasol population under all tested electrode sizes (two-way ANOVA, p < 0.05). Our simulations also suggested that smaller electrodes exhibit a higher range of controllable stimulation parameters to achieve pre-defined performance of RGC excitation. ..

Immersive Virtual Reality Simulations of Bionic Vision

Bionic vision uses neuroprostheses to restore useful vision to people living with incurable blindness. However, a major outstanding challenge is predicting what people "see" when they use their devices. The limited field of view of current devices necessitates head movements to scan the scene, which is difficult to simulate on a computer screen. In addition, many computational models of bionic vision lack biological realism. To address these challenges, we present VR-SPV, an open-source virtual reality toolbox for simulated prosthetic vision that uses a psychophysically validated computational model to allow sighted participants to "see through the eyes" of a bionic eye user. To demonstrate its utility, we systematically evaluated how clinically reported visual distortions affect performance in a letter recognition and an immersive obstacle avoidance task. Our results highlight the importance of using an appropriate phosphene model when predicting visual outcomes for bionic vision.

End-to-end optimization of prosthetic vision

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a nontrivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper, we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments, we show that such an approach is able to automatically find a task-specific stimulation protocol. The results of these proof-of-principle experiments illustrate the potential of end-to-end optimization for prosthetic vision. The presented approach is highly modular and our approach could be extended to automated dynamic optimization of prosthetic vision for everyday tasks, given any specific constraints, accommodating individual requirements of the end-user.

Selective Activation of Retinal Ganglion Cell Subtypes Through Targeted Electrical Stimulation Parameters

To restore vision to the low vision, epiretinal implants have been developed to electrically stimulate the healthy retinal ganglion cells (RGCs) in the degenerate retina. Given the diversity of retinal ganglion cells as well as the difference in their visual function, selective activation of RGCs subtypes can significantly improve the quality of the restored vision. Our recent results demonstrated that with the proper modulation of the current amplitude, small D1-bistratified cells with the contribution to blue/yellow color opponent pathway can be selectively activated at high frequency (200 Hz). The computational results correlated with the clinical findings revealing the blue sensation of 5/7 subjects with epiretinal implants at high frequency. Here we further explored the impacts of alterations in pulse duration and interphase gap on the response of RGCs at high frequency. We used the developed RGCs, A2-monostratified and D1-bistratified, and examined their response to a range of pulse durations (0.1−1.2 ms) and interphase gaps (0−1 ms). We found that the use of short pulse durations with no interphase gap at high frequency increases the differential response of RGCs, offering better opportunities for selective activation of D1 cells. The presence of the interphase gap has shown to reduce the overall differential response of RGCs. We also explored how the low density of calcium channels enhances the responsiveness of RGCs at high frequency.

Transcorneal Electrical Stimulation Induces Long-Lasting Enhancement of Brain Functional and Directional Connectivity in Retinal Degeneration Mice

To investigate neuromodulation of functional and directional connectivity features in both visual and non-visual brain cortices after short-term and long-term retinal electrical stimulation in retinal degeneration mice. We performed spontaneous electrocorticography (ECoG) in retinal degeneration (rd) mice following prolonged transcorneal electrical stimulation (pTES) at varying currents (400, 500 and 600 μA) and different time points (transient or day 1 post-stimulation, 1-week post-stimulation and 2-weeks post-stimulation). We also set up a sham control group of rd mice which did not receive any electrical stimulation. Subsequently we analyzed alterations in cross-frequency coupling (CFC), coherence and directional connectivity of the primary visual cortex and the prefrontal cortex. It was observed that the sham control group did not display any significant changes in brain connectivity across all stages of electrical stimulation. For the stimulated groups, we observed that transient electrical stimulation of the retina did not significantly alter brain coherence and connectivity. However, for 1-week post-stimulation, we identified enhanced increase in theta-gamma CFC. Meanwhile, enhanced coherence and directional connectivity appeared predominantly in theta, alpha and beta oscillations. These alterations occurred in both visual and non-visual brain regions and were dependent on the current amplitude of stimulation. Interestingly, 2-weeks post-stimulation demonstrated long-lasting enhancement in network coherence and connectivity patterns at the level of cross-oscillatory interaction, functional connectivity and directional inter-regional communication between the primary visual cortex and prefrontal cortex. Application of electrical stimulation to the retina evidently neuromodulates brain coherence and connectivity of visual and non-visual cortices in retinal degeneration mice and the observed alterations are largely maintained. pTES holds strong possibility of modulating higher cortical functions including pathways of cognition, awareness, emotion and memory.

Visual-tactile shape perception in the visually restored with artificial vision

Retinal prostheses partially restore vision to late blind patients with retinitis pigmentosa through electrical stimulation of still-viable retinal ganglion cells. We investigated whether the late blind can perform visual–tactile shape matching following the partial restoration of vision via retinal prostheses after decades of blindness.

We tested for visual–visual, tactile–tactile, and visual–tactile two-dimensional shape matching with six Argus II retinal prosthesis patients, ten sighted controls, and eight sighted controls with simulated ultra-low vision. In the Argus II patients, the visual–visual shape matching performance was significantly greater than chance. Although the visual–tactile shape matching performance of the Argus II patients was not significantly greater than chance, it was significantly higher with longer duration of prosthesis use. The sighted controls using natural vision and the sighted controls with simulated ultra-low vision both performed the visual–visual and visual–tactile shape matching tasks significantly more accurately than the Argus II patients. The tactile–tactile matching was not significantly different between the Argus II patients and sighted controls with or without simulated ultra-low vision.

These results show that experienced retinal prosthesis patients can match shapes across the senses and integrate artificial vision with somatosensation. The correlation of retinal prosthesis patients’ crossmodal shape matching performance with the duration of device use supports the value of experience to crossmodal shape learning. These crossmodal shape matching results in Argus II patients are the first step toward understanding crossmodal perception after artificial visual restoration.

Real-Time Optimization of Retinal Ganglion Cell Spatial Activity in Response to Epiretinal Stimulation

Retinal prostheses aim to improve visual perception in patients blinded by photoreceptor degeneration. However, shape and letter perception with these devices is currently limited due to low spatial resolution. Previous research has shown the retinal ganglion cell (RGC) spatial activity and phosphene shapes can vary due to the complexity of retina structure and electrode-retina interactions. Visual percepts elicited by single electrodes differ in size and shapes for different electrodes within the same subject, resulting in interference between phosphenes and an unclear image. Prior work has shown that better patient outcomes correlate with spatially separate phosphenes. In this study we use calcium imaging, in vitro retina, neural networks (NN), and an optimization algorithm to demonstrate a method to iteratively search for optimal stimulation parameters that create focal RGC activation. Our findings indicate that we can converge to stimulation parameters that result in focal RGC activation by sampling less than 1/3 of the parameter space. A similar process implemented clinically can reduce time required for optimizing implant operation and enable personalized fitting of retinal prostheses.

Mechanisms underlying activation of retinal bipolar cells through targeted electrical stimulation: a computational study

Objective. Retinal implants have been developed to electrically stimulate healthy retinal neurons in the progressively degenerated retina. Several stimulation approaches have been proposed to improve the visual percept induced in patients with retinal prostheses. We introduce a computational model capable of simulating the effects of electrical stimulation on retinal neurons. Leveraging this computational platform, we delve into the underlying mechanisms influencing the sensitivity of retinal neurons' response to various stimulus waveforms.Approach. We implemented a model of spiking bipolar cells (BCs) in the magnocellular pathway of the primate retina, diffuse BC subtypes (DB4), and utilized our multiscale admittance method (AM)-NEURON computational platform to characterize the response of BCs to epiretinal electrical stimulation with monophasic, symmetric, and asymmetric biphasic pulses.Main results. Our investigations yielded four notable results: (a) the latency of BCs increases as stimulation pulse duration lengthens; conversely, this latency decreases as the current amplitude increases. (b) Stimulation with a long anodic-first symmetric biphasic pulse (duration > 8 ms) results in a significant decrease in spiking threshold compared to stimulation with similar cathodic-first pulses (from 98.2 to 57.5µA). (c) The hyperpolarization-activated cyclic nucleotide-gated channel was a prominent contributor to the reduced threshold of BCs in response to long anodic-first stimulus pulses. (d) Finally, extending the study to asymmetric waveforms, our results predict a lower BCs threshold using asymmetric long anodic-first pulses compared to that of asymmetric short cathodic-first stimulation.Significance. This study predicts the effects of several stimulation parameters on spiking BCs response to electrical stimulation. Of importance, our findings shed light on mechanisms underlying the experimental observations from the literature, thus highlighting the capability of the methodology to predict and guide the development of electrical stimulation protocols to generate a desired biological response, thereby constituting an ideal testbed for the development of electroceutical devices.

Learning to see again: Perceptual learning of simulated abnormal on- off-cell population responses in sighted individuals

Many forms of artificial sight recovery, such as electronic implants and optogenetic proteins, generally cause simultaneous, rather than complementary firing of on- and off-center retinal cells. Here, using virtual patients—sighted individuals viewing distorted input—we examine whether plasticity might compensate for abnormal neuronal population responses. Five participants were dichoptically presented with a combination of original and contrast-reversed images. Each image (I) and its contrast-reverse (Iʹ) was filtered using a radial checkerboard (F) in Fourier space and its inverse (Fʹ). [I * F′] + [Iʹ * F] was presented to one eye, and [I * F] + [Iʹ * F′] was presented to the other, such that regions of the image that produced on-center responses in one eye produced off-center responses in the other eye, and vice versa. Participants continuously improved in a naturalistic object discrimination task over 20 one-hour sessions. Pre-training and post-training tests suggest that performance improvements were due to two learning processes: learning to recognize objects with reduced visual information and learning to suppress contrast-reversed image information in a non–eye-selective manner. These results suggest that, with training, it may be possible to adapt to the unnatural on- and off-cell population responses produced by electronic and optogenetic sight recovery technologies.

Automatic Identification of Axon Bundle Activation for Epiretinal Prosthesis

Objective:

Retinal prostheses must be able to activate cells in a selective way in order to restore high-fidelity vision. However, inadvertent activation of far-away retinal ganglion cells (RGCs) through electrical stimulation of axon bundles can produce irregular and poorly controlled percepts, limiting artificial vision. In this work, we aim to provide an algorithmic solution to the problem of detecting axon bundle activation with a bi-directional epiretinal prostheses.

Methods:

The algorithm utilizes electrical recordings to determine the stimulation current amplitudes above which axon bundle activation occurs. Bundle activation is defined as the axonal stimulation of RGCs with unknown soma and receptive field locations, typically beyond the electrode array. The method exploits spatiotemporal characteristics of electrically-evoked spikes to overcome the challenge of detecting small axonal spikes.

Results:

The algorithm was validated using large-scale, single-electrode and short pulse, ex vivo stimulation and recording experiments in macaque retina, by comparing algorithmically and manually identified bundle activation thresholds. For 88% of the electrodes analyzed, the threshold identified by the algorithm was within ±10% of the manually identified threshold, with a correlation coefficient of 0.95.

Conclusion:

This works presents a simple, accurate and efficient algorithm to detect axon bundle activation in epiretinal prostheses.

Significance:

The algorithm could be used in a closed-loop manner by a future epiretinal prosthesis to reduce poorly controlled visual percepts associated with bundle activation. Activation of distant cells via axonal stimulation will likely occur in other types of retinal implants and cortical implants, and the method may therefore be broadly applicable.

Spatially patterned bi-electrode epiretinal stimulation for axon avoidance at cellular resolution

Objective.Epiretinal prostheses are designed to restore vision to people blinded by photoreceptor degenerative diseases by stimulating surviving retinal ganglion cells (RGCs), which carry visual signals to the brain. However, inadvertent stimulation of RGCs at their axons can result in non-focal visual percepts, limiting the quality of artificial vision. Theoretical work has suggested that axon activation can be avoided with current stimulation designed to minimize the second spatial derivative of the induced extracellular voltage along the axon. However, this approach has not been verified experimentally at the resolution of single cells.Approach.In this work, a custom multi-electrode array (512 electrodes, 10μm diameter, 60μm pitch) was used to stimulate and record RGCs in macaque retinaex vivoat single-cell, single-spike resolution. RGC activation thresholds resulting from bi-electrode stimulation, which consisted of bipolar currents simultaneously delivered through two electrodes straddling an axon, were compared to activation thresholds from traditional single-electrode stimulation.Main results.On average, across three retinal preparations, the bi-electrode stimulation strategy reduced somatic activation thresholds (∼21%) while increasing axonal activation thresholds (∼14%), thus favoring selective somatic activation. Furthermore, individual examples revealed rescued selective activation of somas that was not possible with any individual electrode.Significance.This work suggests that a bi-electrode epiretinal stimulation strategy can reduce inadvertent axonal activation at cellular resolution, for high-fidelity artificial vision.

A Computational Model of Phosphene Appearance for Epiretinal Prostheses

Retinal neuroprostheses are the only FDA-approved treatment option for blinding degenerative diseases. A major outstanding challenge is to develop a computational model that can accurately predict the elicited visual percepts (phosphenes) across a wide range of electrical stimuli. Here we present a phenomenological model that predicts phosphene appearance as a function of stimulus amplitude, frequency, and pulse duration. The model uses a simulated map of nerve fiber bundles in the retina to produce phosphenes with accurate brightness, size, orientation, and elongation. We validate the model on psychophysical data from two independent studies, showing that it generalizes well to new data, even with different stimuli and on different electrodes. Whereas previous models focused on either spatial or temporal aspects of the elicited phosphenes in isolation, we describe a more comprehensive approach that is able to account for many reported visual effects. The model is designed to be flexible and extensible, and can be fit to data from a specific user. Overall this work is an important first step towards predicting visual outcomes in retinal prosthesis users across a wide range of stimuli.

Modeling ON Cone Bipolar Cells for Electrical Stimulation

Retinal prosthetic systems have been developed to help blind patients suffering from retinal degenerative diseases gain some useful form of vision. Various experimental and computational studies have been performed to test electrical stimulation strategies that can improve the performance of these devices. Detailed computational models of retinal neurons, such as retinal ganglion cells (RGCs) and bipolar cells (BCs), allow us to explore the mechanisms underlying the response of cells to electrical stimulation. While electrophysiological studies have shown the presence of voltage-gated ionic channels in different regions of BCs, many of the existing cone BCs models are assumed to be passive or only contain calcium channels at the synaptic terminals. We have utilized our Admittance Method (AM)-NEURON computational platform to implement a more realistic model of ON-BCs. Our model closely replicates the recent patch-clamp experiments directly measuring the response of ON-BCs to epiretinal electrical stimulation and thereby predicts the regional distributions of the ionic channels. Our computational results further indicate that outward potassium current strongly contributes to the depolarizing voltage transient of ON-BCs in response to electrical stimulation.