Human-in-the-loop optimization of visual prosthetic stimulation

Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

An actor-model framework for visual sensory encoding

A fundamental challenge in neuroengineering is determining a proper artificial input to a sensory system that yields the desired perception. In neuroprosthetics, this process is known as artificial sensory encoding, and it holds a crucial role in prosthetic devices restoring sensory perception in individuals with disabilities. For example, in visual prostheses, one key aspect of artificial image encoding is to downsample images captured by a camera to a size matching the number of inputs and resolution of the prosthesis. Here, we show that downsampling an image using the inherent computation of the retinal network yields better performance compared to learning-free downsampling methods. We have validated a learning-based approach (actor-model framework) that exploits the signal transformation from photoreceptors to retinal ganglion cells measured in explanted mouse retinas. The actor-model framework generates downsampled images eliciting a neuronal response in-silico and ex-vivo with higher neuronal reliability than the one produced by a learning-free approach. During the learning process, the actor network learns to optimize contrast and the kernel's weights. This methodological approach might guide future artificial image encoding strategies for visual prostheses. Ultimately, this framework could be applicable for encoding strategies in other sensory prostheses such as cochlear or limb.

An optimization framework for targeted spinal cord stimulation

Objective. Spinal cord stimulation (SCS) is a common neurostimulation therapy to manage chronic pain. Technological advances have produced new neurostimulation systems with expanded capabilities in an attempt to improve the clinical outcomes associated with SCS. However, these expanded capabilities have dramatically increased the number of possible stimulation parameters and made it intractable to efficiently explore this large parameter space within the context of standard clinical programming procedures. Therefore, in this study, we developed an optimization approach to define the optimal current amplitudes or fractions across individual contacts in an SCS electrode array(s).Approach. We developed an analytic method using the Lagrange multiplier method along with smoothing approximations. To test our optimization framework, we used a hybrid computational modeling approach that consisted of a finite element method model and multi-compartment models of axons and cells within the spinal cord. Moreover, we extended our approach to multi-objective optimization to explore the trade-off between activating regions of interest (ROIs) and regions of avoidance (ROAs).Main results. For simple ROIs, our framework suggested optimized configurations that resembled simple bipolar configurations. However, when we considered multi-objective optimization, our framework suggested nontrivial stimulation configurations that could be selected from Pareto fronts to target multiple ROIs or avoid ROAs.Significance. We developed an optimization framework for targeted SCS. Our method is analytic, which allows for the fast calculation of optimal solutions. For the first time, we provided a multi-objective approach for selective SCS. Through this approach, we were able to show that novel configurations can provide neural recruitment profiles that are not possible with conventional stimulation configurations (e.g. bipolar stimulation). Most importantly, once integrated with computational models that account for sources of interpatient variability (e.g. anatomy, electrode placement), our optimization framework can be utilized to provide stimulation settings tailored to the needs of individual patients.

A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.

Towards aSmart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Objective.How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies.Approach.Rather than aiming to represent the visual scene as naturally as possible, aSmart Bionic Eyecould provide visual augmentations through the means of artificial intelligence-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care.Main results.Complementary to existing research aiming to restore natural vision, we propose a patient-centered approach to incorporate deep learning-based visual augmentations into the next generation of devices.Significance.The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.