Real and virtual mobility performance in simulated prosthetic vision

Photovoltaic, wireless wide-field epiretinal prosthesis to treat retinitis pigmentosa

PURPOSE

To develop and evaluate a photovoltaic, wireless wide-field epiretinal prosthesis for the treatment of retinitis pigmentosa.

METHODS

A mosaic array of thinned silicon-based photodiodes with integrated thin-film stimulation electrodes was fabricated with a flexible polyimide substrate film to form a film-based miniaturized electronic system with wireless optical power and signal transmission and integrated electrostimulation. Manufactured implants were characterized with respect to their optoelectronic performance and biocompatibility following DIN EN ISO 10993.

RESULTS

A 14 mm diameter prosthesis containing 1276 pixels with a maximum sensitivity at a near infrared wavelength of 905 nm and maximized stimulation current density 30-50 μm below the electrodes was developed for direct activation of retinal ganglion cells during epiretinal stimulation. Fabricated prostheses demonstrated mucosal tolerance and the preservation of both metabolic activity, proliferation and membrane integrity of human fibroblasts as well as the retinal functions of bovine retinas. Illumination of the prosthesis, which was placed epiretinally on an isolated perfused bovine retina, with infrared light resulted in electrophysiological recordings reminiscent of an a-wave (hyperpolarization) and b-wave (depolarization).

CONCLUSIONS

A photovoltaic, wireless wide-field epiretinal prosthesis for the treatment of retinitis pigmentosa using near infrared light for signal transmission was designed, manufactured and its biocompatibility and functionality demonstrated in vitro and ex vivo.

Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

A novel simulation paradigm utilising MRI-derived phosphene maps for cortical prosthetic vision

Objective.We developed a realistic simulation paradigm for cortical prosthetic vision and investigated whether we can improve visual performance using a novel clustering algorithm.Approach.Cortical visual prostheses have been developed to restore sight by stimulating the visual cortex. To investigate the visual experience, previous studies have used uniform phosphene maps, which may not accurately capture generated phosphene map distributions of implant recipients. The current simulation paradigm was based on the Human Connectome Project retinotopy dataset and the placement of implants on the cortices from magnetic resonance imaging scans. Five unique retinotopic maps were derived using this method. To improve performance on these retinotopic maps, we enabled head scanning and a density-based clustering algorithm was then used to relocate centroids of visual stimuli. The impact of these improvements on visual detection performance was tested. Using spatially evenly distributed maps as a control, we recruited ten subjects and evaluated their performance across five sessions on the Berkeley Rudimentary Visual Acuity test and the object recognition task.Main results.Performance on control maps is significantly better than on retinotopic maps in both tasks. Both head scanning and the clustering algorithm showed the potential of improving visual ability across multiple sessions in the object recognition task.Significance.The current paradigm is the first that simulates the experience of cortical prosthetic vision based on brain scans and implant placement, which captures the spatial distribution of phosphenes more realistically. Utilisation of evenly distributed maps may overestimate the performance that visual prosthetics can restore. This simulation paradigm could be used in clinical practice when making plans for where best to implant cortical visual prostheses.

Design Optimization and Characterization with Fabrication of Nanomaterials-Based Photo Diode Cell for Subretinal Implant Application

An ultrathin nano photodiode array fabricated in a flexible substrate can be an ideal therapeutic replacement for degenerated photoreceptor cells damaged by Age-related Macula Degeneration (AMD) and Retinitis Pigmentosa (RP), such as retinal infections. Silicon-based photodiode arrays have been attempted as artificial retinas. Considering the difficulties caused by hard silicon subretinal implants, researchers have diverted their attention towards organic photovoltaic cells-based subretinal implants. Indium-Tin Oxide (ITO) has been a favorite choice as an anode electrode. A mix of poly(3-hexylthiophene) and [6,6]-phenyl C61-butyric acid methyleste (P3HT: PCBM) has been utilized as an active layer in such nanomaterial-based subretinal implants. Though encouraging results have been obtained during the trial of such retinal implants, the need to replace ITO with a suitable transparent conductive electrode will be a suitable substitute. Further, conjugated polymers have been used as active layers in such photodiodes and have shown delamination in the retinal space over time despite their biocompatibility. This research attempted to fabricate and characterize Bulk Hetero Junction (BHJ) based Nano Photo Diode (NPD) utilizing Graphene-polyethylene terephthalate (G-PET)/semiconducting Single-Wall Carbon Nano Tubes (s-SWCNT): fullerene (C₆₀) blend/aluminium (Al) structure to determine the issues in the development of subretinal prosthesis. An effective design approach adopted in this analysis has resulted in developing an NPD with an Efficiency of 10.1% in a non-ITO-driven NPD structure. Additionally, the results show that the efficiency can be further improved by increasing active layer thickness.

Towards aSmart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness

Objective.How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies.Approach.Rather than aiming to represent the visual scene as naturally as possible, aSmart Bionic Eyecould provide visual augmentations through the means of artificial intelligence-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care.Main results.Complementary to existing research aiming to restore natural vision, we propose a patient-centered approach to incorporate deep learning-based visual augmentations into the next generation of devices.Significance.The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.

POLYRETINA restores light responses in vivo in blind Göttingen minipigs

Retinal prostheses hold the potential for artificial vision in blind people affected by incurable diseases of the outer retinal layer. Available technologies provide only a small field of view: a significant limitation for totally blind people. To overcome this problem, we recently proposed a large and high-density photovoltaic epiretinal device, known as POLYRETINA. Here, we report the in vivo assessment of POLYRETINA. First, we characterise a model of chemically-induced blindness in Göttingen minipigs. Then, we develop and test a minimally invasive injection procedure to insert the large epiretinal implant into the eye. Last, we show that POLYRETINA restores light-evoked cortical responses in blind animals at safe irradiance levels. These results indicate that POLYRETINA holds the potential for artificial vision in totally blind patients affected by retinitis pigmentosa.

Virtual reality validation of naturalistic modulation strategies to counteract fading in retinal stimulation

OBJECTIVE

Temporal resolution is a key challenge in artificial vision. Several prosthetic approaches are limited by the perceptual fading of evoked phosphenes upon repeated stimulation from the same electrode. Therefore, implanted patients are forced to perform active scanning, via head movements, to refresh the visual field viewed by the camera. However, active scanning is a draining task, and it is crucial to find compensatory strategies to reduce it.

APPROACH

To address this question, we implemented perceptual fading in simulated prosthetic vision using virtual reality. Then, we quantified the effect of fading on two indicators: the time to complete a reading task and the head rotation during the task. We also tested if stimulation strategies previously proposed to increase the persistence of responses in retinal ganglion cells to electrical stimulation could improve these indicators.

MAIN RESULTS

This study shows that stimulation strategies based on interrupted pulse trains and randomisation of the pulse duration allows significant reduction of both the time to complete the task and the head rotation during the task.

SIGNIFICANCE

The stimulation strategy used in retinal implants is crucial to counteract perceptual fading and to reduce active head scanning during prosthetic vision. In turn, less active scanning might improve the patient's comfort in artificial vision.

End-to-end optimization of prosthetic vision

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a nontrivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper, we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments, we show that such an approach is able to automatically find a task-specific stimulation protocol. The results of these proof-of-principle experiments illustrate the potential of end-to-end optimization for prosthetic vision. The presented approach is highly modular and our approach could be extended to automated dynamic optimization of prosthetic vision for everyday tasks, given any specific constraints, accommodating individual requirements of the end-user.

Real-world indoor mobility with simulated prosthetic vision: The benefits and feasibility of contour-based scene simplification at different phosphene resolutions

Neuroprosthetic implants are a promising technology for restoring some form of vision in people with visual impairments via electrical neurostimulation in the visual pathway. Although an artificially generated prosthetic percept is relatively limited compared with normal vision, it may provide some elementary perception of the surroundings, re-enabling daily living functionality. For mobility in particular, various studies have investigated the benefits of visual neuroprosthetics in a simulated prosthetic vision paradigm with varying outcomes. The previous literature suggests that scene simplification via image processing, and particularly contour extraction, may potentially improve the mobility performance in a virtual environment. In the current simulation study with sighted participants, we explore both the theoretically attainable benefits of strict scene simplification in an indoor environment by controlling the environmental complexity, as well as the practically achieved improvement with a deep learning-based surface boundary detection implementation compared with traditional edge detection. A simulated electrode resolution of 26 × 26 was found to provide sufficient information for mobility in a simple environment. Our results suggest that, for a lower number of implanted electrodes, the removal of background textures and within-surface gradients may be beneficial in theory. However, the deep learning-based implementation for surface boundary detection did not improve mobility performance in the current study. Furthermore, our findings indicate that, for a greater number of electrodes, the removal of within-surface gradients and background textures may deteriorate, rather than improve, mobility. Therefore, finding a balanced amount of scene simplification requires a careful tradeoff between informativity and interpretability that may depend on the number of implanted electrodes.

Learning to see again: Perceptual learning of simulated abnormal on- off-cell population responses in sighted individuals

Many forms of artificial sight recovery, such as electronic implants and optogenetic proteins, generally cause simultaneous, rather than complementary firing of on- and off-center retinal cells. Here, using virtual patients—sighted individuals viewing distorted input—we examine whether plasticity might compensate for abnormal neuronal population responses. Five participants were dichoptically presented with a combination of original and contrast-reversed images. Each image (I) and its contrast-reverse (Iʹ) was filtered using a radial checkerboard (F) in Fourier space and its inverse (Fʹ). [I * F′] + [Iʹ * F] was presented to one eye, and [I * F] + [Iʹ * F′] was presented to the other, such that regions of the image that produced on-center responses in one eye produced off-center responses in the other eye, and vice versa. Participants continuously improved in a naturalistic object discrimination task over 20 one-hour sessions. Pre-training and post-training tests suggest that performance improvements were due to two learning processes: learning to recognize objects with reduced visual information and learning to suppress contrast-reversed image information in a non–eye-selective manner. These results suggest that, with training, it may be possible to adapt to the unnatural on- and off-cell population responses produced by electronic and optogenetic sight recovery technologies.

Electrode Dropout Compensation in Visual Prostheses: An Optimal Object Placement Approach

Visual prostheses provide promising solution to the blind through partial restoration of their vision via electrical stimulation of the visual system. However, there are some challenges that hinder the ability of subjects implanted with visual prostheses to correctly identify an object. One of these challenges is electrode dropout; the malfunction of some electrodes resulting in consistently dark phosphenes. In this paper, we propose a dropout handling algorithm for better and faster identification of objects. In this algorithm, phosphenes representing the object are translated to another location within the same image that has the minimum number of dropouts. Using simulated prosthetic vision, experiments were conducted to test the efficacy of our proposed algorithm. Electrode dropout rates of 10%, 20% and 30% were examined. Our results demonstrate significant increase in the object recognition accuracy, reduction in the recognition time and increase in the recognition confidence level using the proposed approach compared to presenting the images without dropout handling.Clinical Relevance- These results demonstrate the utility of dropout handling in enhancing the perception of images in prosthetic vision.

New epiretinal implant with integrated sensor chips for optical capturing shows a good biocompatibility profile in vitro and in vivo

BACKGROUND

Retinal degenerative diseases, e.g., retinitis pigmentosa, cause a severe decline of the visual function up to blindness. Treatment still remains difficult; however, implantation of retinal prostheses can help restoring vision. In this study, the biocompatibility and surgical feasibility of a newly developed epiretinal stimulator (OPTO-EPIRET) was investigated. The previously developed implant was extended by an integrated circuit-based optical capturing, which will enable the immediate conversion of the visual field into stimulation patterns to stimulate retinal ganglion cells.

RESULTS

The biocompatibility of the OPTO-EPIRET was investigated in vitro using the two different cell lines L-929 and R28. Direct and indirect contact were analyzed in terms of cell proliferation, cell viability, and gene expression. The surgical feasibility was initially tested by implanting the OPTO-EPIRET in cadaveric rabbit eyes. Afterwards, inactive devices were implanted in six rabbits for feasibility and biocompatibility testings in vivo. In follow-up controls (1-12 weeks post-surgery), the eyes were examined using fundoscopy and optical coherence tomography. After finalization, histological examination was performed to analyze the retinal structure. Regarding the in vitro biocompatibility, no significant influence on cell viability was detected (L929: < 1.3% dead cells; R-28: < 0.8% dead cells). The surgery, which comprised phacoemulsification, vitrectomy, and implantation of the OPTO-EPIRET through a 9-10 mm corneal incision, was successfully established. The implant was fixated with a retinal tack. Vitreal hemorrhage or retinal tearing occurred as main adverse effects. Transitional corneal edema caused difficulties in post-surgical imaging.

CONCLUSIONS

The OPTO-EPIRET stimulator showed a good biocompatibility profile in vitro. Furthermore, the implantation surgery was shown to be feasible. However, further design optimization steps are necessary to avoid intra- and postoperative complications. Overall, the OPTO-EPIRET will allow for a wide visual field and good visual acuity due to a high density of electrodes in the central retina.

Influence of field of view in visual prostheses design: Analysis with a VR system

OBJECTIVE

Visual prostheses are designed to restore partial functional vision in patients with total vision loss. Retinal visual prostheses provide limited capabilities as a result of low resolution, limited field of view and poor dynamic range. Understanding the influence of these parameters in the perception results can guide prostheses research and design.

APPROACH

In this work, we evaluate the influence of field of view with respect to spatial resolution in visual prostheses, measuring the accuracy and response time in a search and recognition task. Twenty-four normally sighted participants were asked to find and recognize usual objects, such as furniture and home appliance in indoor room scenes. For the experiment, we use a new simulated prosthetic vision system that allows simple and effective experimentation. Our system uses a virtual-reality environment based on panoramic scenes. The simulator employs a head-mounted display which allows users to feel immersed in the scene by perceiving the entire scene all around. Our experiments use public image datasets and a commercial head-mounted display. We have also released the virtual-reality software for replicating and extending the experimentation.

MAIN RESULTS

Results show that the accuracy and response time decrease when the field of view is increased. Furthermore, performance appears to be correlated with the angular resolution, but showing a diminishing return even with a resolution of less than 2.3 phosphenes per degree.

SIGNIFICANCE

Our results seem to indicate that, for the design of retinal prostheses, it is better to concentrate the phosphenes in a small area, to maximize the angular resolution, even if that implies sacrificing field of view.

Harmonization of Outcomes and Vision Endpoints in Vision Restoration Trials: Recommendations from the International HOVER Taskforce

Translational research in vision prosthetics, gene therapy, optogenetics, stem cell and other forms of transplantation, and sensory substitution is creating new therapeutic options for patients with neural forms of blindness. The technical challenges faced by each of these disciplines differ considerably, but they all face the same challenge of how to assess vision in patients with ultra-low vision (ULV), who will be the earliest subjects to receive new therapies. Historically, there were few tests to assess vision in ULV patients. In the 1990s, the field of visual prosthetics expanded rapidly, and this activity led to a heightened need to develop better tests to quantify end points for clinical studies. Each group tended to develop novel tests, which made it difficult to compare outcomes across groups. The common lack of validation of the tests and the variable use of controls added to the challenge of interpreting the outcomes of these clinical studies. In 2014, at the bi-annual International "Eye and the Chip" meeting of experts in the field of visual prosthetics, a group of interested leaders agreed to work cooperatively to develop the International Harmonization of Outcomes and Vision Endpoints in Vision Restoration Trials (HOVER) Taskforce. Under this banner, more than 80 specialists across seven topic areas joined an effort to formulate guidelines for performing and reporting psychophysical tests in humans who participate in clinical trials for visual restoration. This document provides the complete version of the consensus opinions from the HOVER taskforce, which, together with its rules of governance, will be posted on the website of the Henry Ford Department of Ophthalmology (www.artificialvision.org). Research groups or companies that choose to follow these guidelines are encouraged to include a specific statement to that effect in their communications to the public. The Executive Committee of the HOVER Taskforce will maintain a list of all human psychophysical research in the relevant fields of research on the same website to provide an overview of methods and outcomes of all clinical work being performed in an attempt to restore vision to the blind. This website will also specify which scientific publications contain the statement of certification. The website will be updated every 2 years and continue to exist as a living document of worldwide efforts to restore vision to the blind. The HOVER consensus document has been written by over 80 of the world's experts in vision restoration and low vision and provides recommendations on the measurement and reporting of patient outcomes in vision restoration trials.

Semantic and structural image segmentation for prosthetic vision

Prosthetic vision is being applied to partially recover the retinal stimulation of visually impaired people. However, the phosphenic images produced by the implants have very limited information bandwidth due to the poor resolution and lack of color or contrast. The ability of object recognition and scene understanding in real environments is severely restricted for prosthetic users. Computer vision can play a key role to overcome the limitations and to optimize the visual information in the prosthetic vision, improving the amount of information that is presented. We present a new approach to build a schematic representation of indoor environments for simulated phosphene images. The proposed method combines a variety of convolutional neural networks for extracting and conveying relevant information about the scene such as structural informative edges of the environment and silhouettes of segmented objects. Experiments were conducted with normal sighted subjects with a Simulated Prosthetic Vision system. The results show good accuracy for object recognition and room identification tasks for indoor scenes using the proposed approach, compared to other image processing methods.

A model of ganglion axon pathways accounts for percepts elicited by retinal implants

Degenerative retinal diseases such as retinitis pigmentosa and macular degeneration cause irreversible vision loss in more than 10 million people worldwide. Retinal prostheses, now implanted in over 250 patients worldwide, electrically stimulate surviving cells in order to evoke neuronal responses that are interpreted by the brain as visual percepts ('phosphenes'). However, instead of seeing focal spots of light, current implant users perceive highly distorted phosphenes that vary in shape both across subjects and electrodes. We characterized these distortions by asking users of the Argus retinal prosthesis system (Second Sight Medical Products Inc.) to draw electrically elicited percepts on a touchscreen. Using ophthalmic fundus imaging and computational modeling, we show that elicited percepts can be accurately predicted by the topographic organization of optic nerve fiber bundles in each subject's retina, successfully replicating visual percepts ranging from 'blobs' to oriented 'streaks' and 'wedges' depending on the retinal location of the stimulating electrode. This provides the first evidence that activation of passing axon fibers accounts for the rich repertoire of phosphene shape commonly reported in psychophysical experiments, which can severely distort the quality of the generated visual experience. Overall our findings argue for more detailed modeling of biological detail across neural engineering applications.

Advances in retinal prosthesis systems

Retinal prosthesis systems have undergone significant advances in the past quarter century, resulting in the development of several different novel surgical and engineering approaches. Encouraging results have demonstrated partial visual restoration, with improvement in both coarse objective function and performance of everyday tasks. To date, four systems have received marketing approval for use in Europe or the United States, with numerous others undergoing preclinical and clinical evaluation, reflecting the established safety profile of these devices for chronic implantation. This progress represents the first notion that the field of visual restorative medicine could offer blind patients a hope of real and measurable benefit. However, there are numerous complex engineering and biophysical obstacles still to be overcome, to reconcile the gap that remains between artificial and natural vision. Current developments in the form of enhanced image processing algorithms and data transfer approaches, combined with emerging nanofabrication and conductive polymerization techniques, herald an exciting and innovative future for retinal prosthetics. This review provides an update of retinal prosthetic systems currently undergoing development and clinical trials while also addressing future challenges in the field, such as the assessment of functional outcomes in ultra-low vision and strategies for tackling existing hardware and software constraints.

Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision

Purpose

Efficacy of current visual prostheses in object recognition is limited. Among various limitations to be addressed, such as low resolution and low dynamic range, here we focus on reducing the impact of background clutter on object recognition. We have proposed the use of motion parallax via head-mounted camera lateral scanning and computationally stabilizing the object of interest (OI) to support neural background decluttering. Simulations in head-mounted displays (HMD), mimicking the proposed effect, were used to test object recognition in normally sighted subjects.

Methods

Images (24° field of view) were captured from multiple viewpoints and presented at a low resolution (20 × 20). All viewpoints were centered on the OI. Experimental conditions (2 × 3) included clutter (with or without) × head scanning (single viewpoint, 9 coherent viewpoints corresponding to subjects' head positions, and 9 randomly associated viewpoints). Subjects used lateral head movements to view OIs in the HMD. Each object was displayed only once for each subject.

Results

The median recognition rate without clutter was 40% for all head scanning conditions. Performance with synthetic background clutter dropped to 10% in the static condition, but it was improved to 20% with the coherent and random head scanning (corrected P = 0.005 and P = 0.049, respectively).

Conclusions

Background decluttering using motion parallax cues but not the coherent multiple views of the OI improved object recognition in low-resolution images. The improvement did not fully eliminate the impact of background.

Translational Relevance

Motion parallax is an effective but incomplete decluttering solution for object recognition with visual prostheses.

Design and validation of a foldable and photovoltaic wide-field epiretinal prosthesis

Retinal prostheses have been developed to fight blindness in people affected by outer retinal layer dystrophies. To date, few hundred patients have received a retinal implant. Inspired by intraocular lenses, we have designed a foldable and photovoltaic wide-field epiretinal prosthesis (named POLYRETINA) capable of stimulating wireless retinal ganglion cells. Here we show that within a visual angle of 46.3 degrees, POLYRETINA embeds 2215 stimulating pixels, of which 967 are in the central area of 5 mm, it is foldable to allow implantation through a small scleral incision, and it has a hemispherical shape to match the curvature of the eye. We demonstrate that it is not cytotoxic and respects optical and thermal safety standards; accelerated ageing shows a lifetime of at least 2 years. POLYRETINA represents significant progress towards the improvement of both visual acuity and visual field with the same device, a current challenging issue in the field.

An optimized content-aware image retargeting method: toward expanding the perceived visual field of the high-density retinal prosthesis recipients

OBJECTIVE

Retinal prosthesis devices have shown great value in restoring some sight for individuals with profoundly impaired vision, but the visual acuity and visual field provided by prostheses greatly limit recipients' visual experience. In this paper, we employ computer vision approaches to seek to expand the perceptible visual field in patients implanted potentially with a high-density retinal prosthesis while maintaining visual acuity as much as possible.

APPROACH

We propose an optimized content-aware image retargeting method, by introducing salient object detection based on color and intensity-difference contrast, aiming to remap important information of a scene into a small visual field and preserve their original scale as much as possible. It may improve prosthetic recipients' perceived visual field and aid in performing some visual tasks (e.g. object detection and object recognition). To verify our method, psychophysical experiments, detecting object number and recognizing objects, are conducted under simulated prosthetic vision. As control, we use three other image retargeting techniques, including Cropping, Scaling, and seam-assisted shrinkability.

MAIN RESULTS

Results show that our method outperforms in preserving more key features and has significantly higher recognition accuracy in comparison with other three image retargeting methods under the condition of small visual field and low-resolution.

SIGNIFICANCE

The proposed method is beneficial to expand the perceived visual field of prosthesis recipients and improve their object detection and recognition performance. It suggests that our method may provide an effective option for image processing module in future high-density retinal implants.

Recognition of a Virtual Scene via Simulated Prosthetic Vision

In order to effectively aid the blind with optimal low-resolution vision and visual recovery training, pathfinding and recognition tests were performed using a simulated visual prosthetic scene. Simple and complex virtual scenes were built using 3DMAX and Unity, and pixelated to three different resolutions (32 × 32, 64 × 64, and 128 × 128) for real-time pixel processing. Twenty subjects were recruited to complete the pathfinding and object recognition tasks within the scene. The recognition accuracy and time required were recorded and analyzed after the trials. In the simple simulated prosthetic vision (SPV) scene, when the resolution was increased from 32 × 32 to 48 × 48, the object recognition time decreased from 92.19 ± 6.97 to 43.05 ± 6.08 s, and the recognition accuracy increased from 51.22 ± 8.53 to 85.52 ± 4.93%. Furthermore, the number of collisions decreased from 10.00 ± 2.31 to 3.00 ± 0.68. When the resolution was increased from 48 × 48 to 64 × 64, the object recognition time further decreased from 43.05 ± 6.08 to 19.46 ± 3.71 s, the recognition accuracy increased from 85.52 ± 4.93 to 96.89 ± 2.06%, and the number of collisions decreased from 3.00 ± 0.68 to 1.00 ± 0.29. In complex scenes, the time required to recognize the room type decreased from 115.00 ± 23.02 to 68.25 ± 17.23 s, and object recognition accuracy increased from 65.69 ± 9.61 to 80.42 ± 7.70% when the resolution increased from 48 × 48 to 64 × 64. When the resolution increased from 64 × 64 to 128 × 128, the time required to recognize the room type decreased from 68.25 ± 17.23 to 44.88 ± 9.94 s, and object recognition accuracy increased from 80.42 ± 7.71 to 85.69 ± 7.39%. Therefore, one can conclude that there are correlations between pathfinding and recognition. When the resolution increased, the time required for recognition decreased, the recognition accuracy increased, and the number of collisions decreased. Although the subjects could partially complete the recognition task at a resolution of 32 × 32, the recognition time was too long and recognition accuracy was not good enough to identify simple scenes. Complex scenes required a resolution of at least 48 × 48 for complete recognition. In addition, increasing the resolution shortened the time required to identify the type of room, and improved the recognition accuracy.

Simplification of Visual Rendering in Simulated Prosthetic Vision Facilitates Navigation

Visual neuroprostheses are still limited and simulated prosthetic vision (SPV) is used to evaluate potential and forthcoming functionality of these implants. SPV has been used to evaluate the minimum requirement on visual neuroprosthetic characteristics to restore various functions such as reading, objects and face recognition, object grasping, etc. Some of these studies focused on obstacle avoidance but only a few investigated orientation or navigation abilities with prosthetic vision. The resolution of current arrays of electrodes is not sufficient to allow navigation tasks without additional processing of the visual input. In this study, we simulated a low resolution array (15 × 18 electrodes, similar to a forthcoming generation of arrays) and evaluated the navigation abilities restored when visual information was processed with various computer vision algorithms to enhance the visual rendering. Three main visual rendering strategies were compared to a control rendering in a wayfinding task within an unknown environment. The control rendering corresponded to a resizing of the original image onto the electrode array size, according to the average brightness of the pixels. In the first rendering strategy, vision distance was limited to 3, 6, or 9 m, respectively. In the second strategy, the rendering was not based on the brightness of the image pixels, but on the distance between the user and the elements in the field of view. In the last rendering strategy, only the edges of the environments were displayed, similar to a wireframe rendering. All the tested renderings, except the 3 m limitation of the viewing distance, improved navigation performance and decreased cognitive load. Interestingly, the distance-based and wireframe renderings also improved the cognitive mapping of the unknown environment. These results show that low resolution implants are usable for wayfinding if specific computer vision algorithms are used to select and display appropriate information regarding the environment.

Laser fabrication of electrical feedthroughs in polymer encapsulations for active implantable medical devices

Hermetic electrical feedthroughs are essential for safe and functional active implantable biomedical devices and for a wide range of other applications such as batteries, supercapacitors, OLEDs and solar cells. Ceramics and metals have previously been the materials of choice for encapsulations, while polymers have advantages of ease of mass production and end user compatibility. We demonstrate a laser sealing technology that gives hermetic, mechanically strong feedthroughs with low electrical resistance in a polyetheretherketone (PEEK) encapsulation. The conductive pathways are wires and sputtered thin films. The water vapor transmission rate through the fabricated encapsulations is comparable to that of PEEK itself.

Neuroprosthetics in amputee and brain injury rehabilitation

The goals of rehabilitation medicine programs are to promote health, restore functional impairments and improve quality of life. The field of neuroprosthetics has evolved over the last decade given an improved understanding of neuroscience and the incorporation of advanced biotechnology and neuroengineering in the rehabilitation setting to develop adaptable applications to help facilitate recovery for individuals with amputations and brain injury. These applications may include a simple cognitive prosthetics aid for impaired memory in brain-injured individuals to myoelectric prosthetics arms with artificial proprioceptive feedback for those with upper extremity amputations. The integration of neuroprosthetics into the existing framework of current rehabilitation approaches not only improves quality-of-care and outcomes but help broadens current rehabilitation treatment paradigms. Although, we are in the infancy of the understanding the true benefit of neuroprosthetics and its clinical applications in the rehabilitation setting there is tremendous amount of promise for future research and development of tools to help facilitate recovery and improve quality of life in individuals with disabilities.

Assistive peripheral phosphene arrays deliver advantages in obstacle avoidance in simulated end-stage retinitis pigmentosa: a virtual-reality study

OBJECTIVE

The prospective efficacy of peripheral retinal prostheses for guiding orientation and mobility in the absence of residual vision, as compared to an implant for the central visual field (VF), was evaluated using simulated prosthetic vision (SPV).

APPROACH

Sighted volunteers wearing a head-mounted display performed an obstacle circumvention task under SPV. Mobility and orientation performance with three layouts of prosthetic vision were compared: peripheral prosthetic vision of higher visual acuity (VA) but limited VF, of wider VF but limited VA, as well as centrally restricted prosthetic vision. Learning curves using these layouts were compared fitting an exponential model to the mobility and orientation measures.

MAIN RESULTS

Using peripheral layouts, performance was superior to the central layout. Walking speed with both higher-acuity and wider-angle layouts was 5.6% higher, and mobility errors reduced by 46.4% and 48.6%, respectively, as compared to the central layout. The wider-angle layout yielded the least number of collisions, 63% less than the higher-acuity and 73% less than the central layout. Using peripheral layouts, the number of visual-scanning related head movements was 54.3% (higher-acuity) and 60.7% (wider-angle) lower, as compared to the central layout, and the ratio of time standing versus time walking was 51.9% and 61.5% lower, respectively. Learning curves did not differ between layouts, except for time standing versus time walking, where both peripheral layouts achieved significantly lower asymptotic values compared to the central layout.

SIGNIFICANCE

Beyond complementing residual vision for an improved performance, peripheral prosthetic vision can effectively guide mobility in the later stages of retinitis pigmentosa (RP) without residual vision. Further, the temporal dynamics of learning peripheral and central prosthetic vision are similar. Therefore, development of a peripheral retinal prosthesis and early implantation to alleviate VF constriction in RP should be considered to extend the target group and the time of benefit for potential retinal prosthesis implantees.

Retinal prostheses: progress toward the next generation implants

In the last decade, various clinical trials proved the capability of visual prostheses, in particular retinal implants, to restore a useful form of vision. These encouraging results promoted the emerging of several strategies for neuronal stimulation aiming at the restoration of sight. Besides the traditional approach based on electrical stimulation through metal electrodes in the different areas of the visual path (e.g., the visual cortex, the lateral geniculate nucleus, the optic nerve, and the retina), novel concepts for neuronal stimulation have been mostly exploited as building blocks of the next generation of retinal implants. This review is focused on critically discussing recent major advancements in the field of retinal stimulation with particular attention to the findings in the application of novel concepts and materials. Last, the major challenges in the field and their clinical implications will be outlined.

Enhancing scene structure in prosthetic vision using iso-disparity contour perturbance maps

We present a novel approach for enhancing structurally significant features in a scene to facilitate safe mobility with prosthetic vision. Previous approaches rely on visually salient features (e.g., intensity gradients, size, texture), or surface fitting (e.g., ground plane extraction), to determine and convey regions of structural change in the scene. Such approaches can be costly to compute, and/or are not guaranteed to detect all features relevant to the needs of safe mobility (e.g., small, low-contrast trip hazards). Assuming a dense disparity image, we propose a novel feature using iso-disparity contours. Regions of significant structural change are detected via a cost function based on local comparisons of iso-disparity contour orientations. Through this, structurally interesting features such as surface boundaries and general clutter are extracted and emphasised in the output visual representation. Our approach is real-time, and requires no surface fitting. Experimental results quantitatively and qualitatively validate our approach.

Towards photorealistic and immersive virtual-reality environments for simulated prosthetic vision: integrating recent breakthroughs in consumer hardware and software

Simulated prosthetic vision (SPV) in normally sighted subjects is an established way of investigating the prospective efficacy of visual prosthesis designs in visually guided tasks such as mobility. To perform meaningful SPV mobility studies in computer-based environments, a credible representation of both the virtual scene to navigate and the experienced artificial vision has to be established. It is therefore prudent to make optimal use of existing hardware and software solutions when establishing a testing framework. The authors aimed at improving the realism and immersion of SPV by integrating state-of-the-art yet low-cost consumer technology. The feasibility of body motion tracking to control movement in photo-realistic virtual environments was evaluated in a pilot study. Five subjects were recruited and performed an obstacle avoidance and wayfinding task using either keyboard and mouse, gamepad or Kinect motion tracking. Walking speed and collisions were analyzed as basic measures for task performance. Kinect motion tracking resulted in lower performance as compared to classical input methods, yet results were more uniform across vision conditions. The chosen framework was successfully applied in a basic virtual task and is suited to realistically simulate real-world scenes under SPV in mobility research. Classical input peripherals remain a feasible and effective way of controlling the virtual movement. Motion tracking, despite its limitations and early state of implementation, is intuitive and can eliminate between-subject differences due to familiarity to established input methods.

Simulated prosthetic vision: improving text accessibility with retinal prostheses

Image processing can improve significantly the every-day life of blind people wearing current and upcoming retinal prostheses relying on an external camera. We propose to use a real-time text localization algorithm to improve text accessibility. An augmented text-specific rendering based on automatic text localization has been developed. It has been evaluated in comparison to the classical rendering through a Simulated Prosthetic Vision (SPV) experiment with 16 subjects. Subjects were able to detect text in natural scenes much faster and further with the augmented rendering compared to the control rendering. Our results show that current and next generation of low resolution retinal prostheses may benefit from real-time text detection algorithms.

Wayfinding with simulated prosthetic vision: performance comparison with regular and structure-enhanced renderings

In this study, we used a simulation of upcoming low-resolution visual neuroprostheses to evaluate the benefit of embedded computer vision techniques in a wayfinding task. We showed that augmenting the classical phosphene rendering with the basic structure of the environment - displaying the ground plane with a different level of brightness - increased both wayfinding performance and cognitive mapping. In spite of the low resolution of current and upcoming visual implants, the improvement of these cognitive functions may already be possible with embedded artificial vision algorithms.

Activation and inhibition of retinal ganglion cells in response to epiretinal electrical stimulation: a computational modelling study

OBJECTIVE

Retinal prosthetic devices aim to restore sight in visually impaired people by means of electrical stimulation of surviving retinal ganglion cells (RGCs). This modelling study aims to demonstrate that RGC inhibition caused by high-intensity cathodic pulses greatly influences their responses to epiretinal electrical stimulation and to investigate the impact of this inhibition on spatial activation profiles as well as their implications for retinal prosthetic device design. Another aim is to take advantage of this inhibition to reduce axonal activation in the nerve fibre layer.

APPROACH

A three-dimensional finite-element model of epiretinal electrical stimulation was utilized to obtain RGC activation and inhibition threshold profiles for a range of parameters.

MAIN RESULTS

RGC activation and inhibition thresholds were highly dependent on cell and stimulus parameters. Activation thresholds were 1.5, 3.4 and 11.3 μA for monopolar electrodes with 5, 20 and 50 μm radii, respectively. Inhibition to activation threshold ratios were mostly within the range 2-10. Inhibition significantly altered spatial patterns of RGC activation. With concentric electrodes and appropriately high levels of stimulus amplitudes, activation of passing axons was greatly reduced.

SIGNIFICANCE

RGC inhibition significantly impacts their spatial activation profiles, and therefore it most likely influences patterns of perceived phosphenes induced by retinal prosthetic devices. Thus this inhibition should be taken into account in future studies concerning retinal prosthesis development. It might be possible to utilize this inhibitory effect to bypass activation of passing axons and selectively stimulate RGCs near their somas and dendrites to achieve more localized phosphenes.

Quasi-monopolar electrical stimulation of the retina: a computational modelling study

OBJECTIVE

In this study we investigated the feasibility of quasi-monopolar (QMP) electrical stimulation for retinal implant devices, using a computational model of the retinal ganglion cell layer.

APPROACH

When used with hexagonally arrayed multiple electrodes, QMP stimulation is a hybrid of hexapolar and conventional monopolar stimulus modes. In hexapolar mode, each active electrode is surrounded by six guards which collectively return the stimulus current, whereas in monopolar mode the injected stimulus current is returned through a distant return electrode. The QMP paradigm, on the other hand, distributes the return current between the guard electrodes as well as the distant return. The electrodes tested were 25, 50 and 100 µm in diameter, with hexagonally arranged centre-to-centre spacing of either double or quadruple this diameter.

MAIN RESULTS

Simulation results indicated that electrode size had minimal effects on subretinal threshold currents, whilst electrode configuration and centre-to-centre spacing played major roles in determining thresholds and spatial activation patterns. Threshold charge densities for 50 and 100 µm electrodes were generally within the safe limit.

SIGNIFICANCE

We found that QMP stimulation offers greater advantages compared to monopolar and hexapolar stimulation, in that it combines the low thresholds of monopolar stimulation with the localized spatial activation achieved with hexapolar electrodes during parallel stimulation.

Recognition of objects in simulated irregular phosphene maps for an epiretinal prosthesis

Visual prostheses offer a possibility of restoring vision to the blind. It is necessary to determine minimum requirements for daily visual tasks. To investigate the recognition of common objects in daily life based on the simulated irregular phosphene maps, the effect of four parameters (resolution, distortion, dropout percentage, and gray scale) on object recognition was investigated. The results showed that object recognition accuracy significantly increased with an increase of resolution. Distortion and dropout percentage had significant impact on the object recognition; with the increase of distortion level and dropout percentage, the recognition decreased considerably. The accuracy decreased significantly only at gray level 2, whereas the other three gray levels showed no obvious difference. The two image processing methods (merging pixels to lower the resolution and edge extraction before lowering resolution) showed significant difference on the object recognition when there was a high degree of distortion level or dot dropout.

A system verification platform for high-density epiretinal prostheses

Retinal prostheses have restored light perception to people worldwide who have poor or no vision as a consequence of retinal degeneration. To advance the quality of visual stimulation for retinal implant recipients, a higher number of stimulation channels is expected in the next generation retinal prostheses, which poses a great challenge to system design and verification. This paper presents a system verification platform dedicated to the development of retinal prostheses. The system includes primary processing, dual-band power and data telemetry, a high-density stimulator array, and two methods for output verification. End-to-end system validation and individual functional block characterization can be achieved with this platform through visual inspection and software analysis. Custom-built software running on the computers also provides a good way for testing new features before they are realized by the ICs. Real-time visual feedbacks through the video displays make it easy to monitor and debug the system. The characterization of the wireless telemetry and the demonstration of the visual display are reported in this paper using a 256-channel retinal prosthetic IC as an example.

Substituting depth for intensity and real-time phosphene rendering: visual navigation under low vision conditions

Navigation and way finding including obstacle avoidance is difficult when visual perception is limited to low resolution, such as is currently available on a bionic eye. Depth visualisation may be a suitable alternative. Such an approach can be evaluated using simulated phosphenes with a wearable mobile virtual reality kit. In this paper, we present two novel approaches: (i) an implementation of depth visualisation; and, (ii) novel methods for rapid rendering of simulated phosphenes with an empirical comparison between them. Our new software-based method for simulated phosphene rendering shows large speed improvements, facilitating the display in real-time of a large number of phosphenes with size and brightness dependent on pixel intensity, and with customised output dynamic range. Further, we describe the protocol, navigation environment and system used for visual navigation experiments to evaluate the use of depth on low resolution simulations of a bionic eye perceptual experience. Results for these experiments show that a depth-based representation is effective for navigation, and shows significant advantages over intensity-based approaches when overhanging obstacles are present. The results of the experiments were reported in [1], [2].

Time-to-contact maps for navigation with a low resolution visual prosthesis

The perception of independently moving objects in the scene is an important capability for prosthetic vision, but is impeded by the limited resolution and dynamic range of current and near-term retinal prostheses. We propose a novel, biologically-inspired visual representation for prosthetic vision based on the recovery of time-to-contact (τ) with surfaces in the scene. The representation directly encodes the extent of motion towards the observer, placing greatest emphasis on objects posing an imminent threat of collision. Our results suggest the proposed τ-based representation may facilitate earlier perception of incoming objects, and provide clearer distinction between moving objects and the static structure of the scene compared with intensity and depth-based scene representations.

Estimation of simulated phosphene size based on tactile perception

Clinical trials have successfully shown that a visual prosthesis can elicit visual perception (phosphenes) in the visual field. Psychophysical studies based on simulated prosthetic vision offer an effective means to evaluate and refine prosthetic vision. We designed three experiments to examine the effect of phosphene luminance, flicker rate, and eccentricity on the ability to estimate simulated phosphene sizes using tactile perception. Thirty subjects participated in the three experiments. There was a linear increase in reported size as visual stimulus size increased. Judgment was significantly affected by stimulus luminance and eccentricity (P < 0.05) but not by flicker rates. Brighter stimuli were perceived as being larger, and the more eccentric the position, the larger the estimated size. These simulation studies, although idealized, suggested that tactile perception is a potential way to estimate phosphene sizes.