A processing platform for optoelectronic/optogenetic retinal prosthesis

Pre-processing visual scenes for retinal prosthesis systems: A comprehensive review

BACKGROUND

Retinal prostheses offer hope for individuals with degenerative retinal diseases by stimulating the remaining retinal cells to partially restore their vision. This review delves into the current advancements in retinal prosthesis technology, with a special emphasis on the pivotal role that image processing and machine learning techniques play in this evolution.

METHODS

We provide a comprehensive analysis of the existing implantable devices and optogenetic strategies, delineating their advantages, limitations, and challenges in addressing complex visual tasks. The review extends to various image processing algorithms and deep learning architectures that have been implemented to enhance the functionality of retinal prosthetic devices. We also illustrate the testing results by demonstrating the clinical trials or using Simulated Prosthetic Vision (SPV) through phosphene simulations, which is a critical aspect of simulating visual perception for retinal prosthesis users.

RESULTS

Our review highlights the significant progress in retinal prosthesis technology, particularly its capacity to augment visual perception among the visually impaired. It discusses the integration between image processing and deep learning, illustrating their impact on individual interactions and navigations within the environment through applying clinical trials and also illustrating the limitations of some techniques to be used with current devices, as some approaches only use simulation even on sighted-normal individuals or rely on qualitative analysis, where some consider realistic perception models and others do not.

CONCLUSION

This interdisciplinary field holds promise for the future of retinal prostheses, with the potential to significantly enhance the quality of life for individuals with retinal prostheses. Future research directions should pivot towards optimizing phosphene simulations for SPV approaches, considering the distorted and confusing nature of phosphene perception, thereby enriching the visual perception provided by these prosthetic devices. This endeavor will not only improve navigational independence but also facilitate a more immersive interaction with the environment.

An actor-model framework for visual sensory encoding

A fundamental challenge in neuroengineering is determining a proper artificial input to a sensory system that yields the desired perception. In neuroprosthetics, this process is known as artificial sensory encoding, and it holds a crucial role in prosthetic devices restoring sensory perception in individuals with disabilities. For example, in visual prostheses, one key aspect of artificial image encoding is to downsample images captured by a camera to a size matching the number of inputs and resolution of the prosthesis. Here, we show that downsampling an image using the inherent computation of the retinal network yields better performance compared to learning-free downsampling methods. We have validated a learning-based approach (actor-model framework) that exploits the signal transformation from photoreceptors to retinal ganglion cells measured in explanted mouse retinas. The actor-model framework generates downsampled images eliciting a neuronal response in-silico and ex-vivo with higher neuronal reliability than the one produced by a learning-free approach. During the learning process, the actor network learns to optimize contrast and the kernel's weights. This methodological approach might guide future artificial image encoding strategies for visual prostheses. Ultimately, this framework could be applicable for encoding strategies in other sensory prostheses such as cochlear or limb.

Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction

Objective. Retinal prostheses are promising devices to restore vision for patients with severe age-related macular degeneration or retinitis pigmentosa disease. The visual processing mechanism embodied in retinal prostheses play an important role in the restoration effect. Its performance depends on our understanding of the retina's working mechanism and the evolvement of computer vision models. Recently, remarkable progress has been made in the field of processing algorithm for retinal prostheses where the new discovery of the retina's working principle and state-of-the-arts computer vision models are combined together.Approach. We investigated the related research on artificial intelligence techniques for retinal prostheses. The processing algorithm in these studies could be attributed to three types: computer vision-related methods, biophysical models, and deep learning models.Main results. In this review, we first illustrate the structure and function of the normal and degenerated retina, then demonstrate the vision rehabilitation mechanism of three representative retinal prostheses. It is necessary to summarize the computational frameworks abstracted from the normal retina. In addition, the development and feature of three types of different processing algorithms are summarized. Finally, we analyze the bottleneck in existing algorithms and propose our prospect about the future directions to improve the restoration effect.Significance. This review systematically summarizes existing processing models for predicting the response of the retina to external stimuli. What's more, the suggestions for future direction may inspire researchers in this field to design better algorithms for retinal prostheses.

Innovative Technologies for Optimized Artificial Vision

Despite significant advances in the treatment of severe eye diseases, certain forms of blindness cannot be cured or improved to this day. These include, for example, retinitis pigmentosa, a hereditary degeneration of photoreceptors. Technology approaches with implantable visual prostheses based on electrical stimulation of remaining neurons in the retina or cortex, have already been tested in a number of patients with limited results. New findings in the biology of these diseases as well as new technological developments give hope for better results in the future.

Retinal Processing: Insights from Mathematical Modelling

The retina is the entrance of the visual system. Although based on common biophysical principles, the dynamics of retinal neurons are quite different from their cortical counterparts, raising interesting problems for modellers. In this paper, I address some mathematically stated questions in this spirit, discussing, in particular: (1) How could lateral amacrine cell connectivity shape the spatio-temporal spike response of retinal ganglion cells? (2) How could spatio-temporal stimuli correlations and retinal network dynamics shape the spike train correlations at the output of the retina? These questions are addressed, first, introducing a mathematically tractable model of the layered retina, integrating amacrine cells' lateral connectivity and piecewise linear rectification, allowing for computing the retinal ganglion cells receptive field together with the voltage and spike correlations of retinal ganglion cells resulting from the amacrine cells networks. Then, I review some recent results showing how the concept of spatio-temporal Gibbs distributions and linear response theory can be used to characterize the collective spike response to a spatio-temporal stimulus of a set of retinal ganglion cells, coupled via effective interactions corresponding to the amacrine cells network. On these bases, I briefly discuss several potential consequences of these results at the cortical level.

Retinal Prosthetic Approaches to Enhance Visual Perception for Blind Patients

Retinal prostheses are implantable devices that aim to restore the vision of blind patients suffering from retinal degeneration, mainly by artificially stimulating the remaining retinal neurons. Some retinal prostheses have successfully reached the stage of clinical trials; however, these devices can only restore vision partially and remain insufficient to enable patients to conduct everyday life independently. The visual acuity of the artificial vision is limited by various factors from both engineering and physiological perspectives. To overcome those issues and further enhance the visual resolution of retinal prostheses, a variety of retinal prosthetic approaches have been proposed, based on optimization of the geometries of electrode arrays and stimulation pulse parameters. Other retinal stimulation modalities such as optics, ultrasound, and magnetics have also been utilized to address the limitations in conventional electrical stimulation. Although none of these approaches have been clinically proven to fully restore the function of a degenerated retina, the extensive efforts made in this field have demonstrated a series of encouraging findings for the next generation of retinal prostheses, and these could potentially enhance the visual acuity of retinal prostheses. In this article, a comprehensive and up-to-date overview of retinal prosthetic strategies is provided, with a specific focus on a quantitative assessment of visual acuity results from various retinal stimulation technologies. The aim is to highlight future directions toward high-resolution retinal prostheses.

Recent Advances on Thermal Management of Flexible Inorganic Electronics

Flexible inorganic electronic devices (FIEDs) consisting of functional inorganic components on a soft polymer substrate have enabled many novel applications such as epidermal electronics and wearable electronics, which cannot be realized through conventional rigid electronics. The low thermal dissipation capacity of the soft polymer substrate of FIEDs demands proper thermal management to reduce the undesired thermal influences. The biointegrated applications of FIEDs pose even more stringent requirements on thermal management due to the sensitive nature of biological tissues to temperature. In this review, we take microscale inorganic light-emitting diodes (μ-ILEDs) as an example of functional components to summarize the recent advances on thermal management of FIEDs including thermal analysis, thermo-mechanical analysis and thermal designs of FIEDs with and without biological tissues. These results are very helpful to understand the underlying heat transfer mechanism and provide design guidelines to optimize FIEDs in practical applications.

Optogenetic Stimulation for Restoring Vision to Patients Suffering From Retinal Degenerative Diseases: Current Strategies and Future Directions

Optogenetic strategies for vision restoration involve photosensitizing surviving retinal neurons following retinal degeneration, using emerging optogenetic techniques. This approach opens the door to a minimally-invasive retinal vision restoration approach. Moreover, light stimulation has the potential to offer better spatial and temporal resolution than conventional retinal electrical prosthetics. Although proof-of-concept studies in animal models have demonstrated the possibility of restoring vision using optogenetic techniques, and initial clinical trials are underway, there are still hurdles to pass before such an approach restores naturalistic vision in humans. One limitation is the development of light stimulation devices to activate optogenetic channels in the retina. Here we review recent progress in the design and implementation of optogenetic stimulation devices and outline the corresponding technological challenges. Finally, while most work to date has focused on providing therapy to patients suffering from retinitis pigmentosa, we provide additional insights into strategies for applying optogenetic vision restoration to patients suffering from age-related macular degeneration.

Real-Time Retinal Processing for High-Resolution Optogenetic Stimulation Device

We present in this paper an image processing technique called Circular Distortion and Motion Compensation (CDMC) that can perform real-time retinal processing with geometric compensation for the ring structure arraignment in the retina for bipolar and ganglion cells. The system was running on an embedded platform of Raspberry Pi 3 and managed to achieve a respectable 12 frames per second on a $640\times 480$ resolution live video capture from a webcam. The system emulates biological processes occurring in the retina such as motion estimation and temporal filtering while compensating for the radial shift of ganglion and bipolar cells in human retina. The proposed algorithm is efficient enough to run on mobile hardware with battery powered device in real-time and it is ideal for high resolution optogenetic stimulation devices that targets the retina.

A head mounted device stimulator for optogenetic retinal prosthesis

Objective. Our main objective is to demonstrate that compact high radiance gallium nitride displays can be used with conventional virtual reality optics to stimulate an optogenetic retina. Hence, we aim to introduce a non-invasive approach to restore vision for people with conditions such as retinitis pigmentosa where there is a remaining viable communication link between the retina and the visual cortex. Approach. We design and implement the headset using a high-density µLED matrix, Raspberry Pi, microcontroller from NXP and virtual reality lens. Then, a test platform is developed to evaluate the performance of the headset and the optical system. Furthermore, image simplification algorithms are used to simplify the scene to be sent to the retina. Moreover, in vivo evaluation of the genetically modified retina response at different light intensity is discussed to prove the reliability of the proposed system. Main results. We demonstrate that in keeping with regulatory guidance, the headset displays need to limit their luminance to 90 kcd m⁻². We demonstrate an optical system with 5.75% efficiency which allows for 0.16 mW mm⁻² irradiance on the retina within the regulatory guidance, but which is capable of an average peak irradiance of 1.35 mW mm⁻². As this is lower than the commonly accepted threshold for channelrhodopsin-2, we demonstrate efficacy through an optical model of an eye onto a biological retina. Significance. We demonstrate a fully functional 8100-pixel headset system including software/hardware which can operate on a standard consumer battery for periods exceeding a 24 h recharge cycle. The headset is capable of delivering enough light to stimulate the genetically modified retina cells and also keeping the amount of light below the regulation threshold for safety.

Extraspectral Imaging for Improving the Perceived Information Presented in Retinal Prosthesis

Retinal prosthesis is steadily improving as a clinical treatment for blindness caused by retinitis pigmentosa. However, despite the continued exciting progress, the level of visual return is still very poor. It is also unlikely that those utilising these devices will stop being legally blind in the near future. Therefore, it is important to develop methods to maximise the transfer of useful information extracted from the visual scene. Such an approach can be achieved by digitally suppressing less important visual features and textures within the scene. The result can be interpreted as a cartoon-like image of the scene. Furthermore, utilising extravisual wavelengths such as infrared can be useful in the decision process to determine the optimal information to present. In this paper, we, therefore, present a processing methodology that utilises information extracted from the infrared spectrum to assist in the preprocessing of the visual image prior to conversion to retinal information. We demonstrate how this allows for enhanced recognition and how it could be implemented for optogenetic forms of retinal prosthesis. The new approach has been quantitatively evaluated on volunteers showing 112% enhancement in recognizing objects over normal approaches.

And Then There Was Light: Perspectives of Optogenetics for Deep Brain Stimulation and Neuromodulation

Deep Brain Stimulation (DBS) has evolved into a well-accepted add-on treatment for patients with severe Parkinsons disease as well as for other chronic neurological conditions. The focal action of electrical stimulation can yield better responses and it exposes the patient to fewer side effects compared to pharmaceuticals distributed throughout the body toward the brain. On the other hand, the current practice of DBS is hampered by the relatively coarse level of neuromodulation achieved. Optogenetics, in contrast, offers the perspective of much more selective actions on the various physiological structures, provided that the stimulated cells are rendered sensitive to the action of light. Optogenetics has experienced tremendous progress since its first in vivo applications about 10 years ago. Recent advancements of viral vector technology for gene transfer substantially reduce vector-associated cytotoxicity and immune responses. This brings about the possibility to transfer this technology into the clinic as a possible alternative to DBS and neuromodulation. New paths could be opened toward a rich panel of clinical applications. Some technical issues still limit the long term use in humans but realistic perspectives quickly emerge. Despite a rapid accumulation of observations about patho-physiological mechanisms, it is still mostly serendipity and empiric adjustments that dictate clinical practice while more efficient logically designed interventions remain rather exceptional. Interestingly, it is also very much the neuro technology developed around optogenetics that offers the most promising tools to fill in the existing knowledge gaps about brain function in health and disease. The present review examines Parkinson's disease and refractory epilepsy as use cases for possible optogenetic stimulation therapies.

An optimized content-aware image retargeting method: toward expanding the perceived visual field of the high-density retinal prosthesis recipients

OBJECTIVE

Retinal prosthesis devices have shown great value in restoring some sight for individuals with profoundly impaired vision, but the visual acuity and visual field provided by prostheses greatly limit recipients' visual experience. In this paper, we employ computer vision approaches to seek to expand the perceptible visual field in patients implanted potentially with a high-density retinal prosthesis while maintaining visual acuity as much as possible.

APPROACH

We propose an optimized content-aware image retargeting method, by introducing salient object detection based on color and intensity-difference contrast, aiming to remap important information of a scene into a small visual field and preserve their original scale as much as possible. It may improve prosthetic recipients' perceived visual field and aid in performing some visual tasks (e.g. object detection and object recognition). To verify our method, psychophysical experiments, detecting object number and recognizing objects, are conducted under simulated prosthetic vision. As control, we use three other image retargeting techniques, including Cropping, Scaling, and seam-assisted shrinkability.

MAIN RESULTS

Results show that our method outperforms in preserving more key features and has significantly higher recognition accuracy in comparison with other three image retargeting methods under the condition of small visual field and low-resolution.

SIGNIFICANCE

The proposed method is beneficial to expand the perceived visual field of prosthesis recipients and improve their object detection and recognition performance. It suggests that our method may provide an effective option for image processing module in future high-density retinal implants.

High Density, High Radiance $\mu$ LED Matrix for Optogenetic Retinal Prostheses and Planar Neural Stimulation

Optical neuron stimulation arrays are important for both in-vitro biology and retinal prosthetic biomedical applications. Hence, in this work, we present an 8100 pixel high radiance photonic stimulator. The chip module vertically combines custom made gallium nitride μ LEDs with a CMOS application specific integrated circuit. This is designed with active pixels to ensure random access and to allow continuous illumination of all required pixels. The μLEDs have been assembled on the chip using a solder ball flip-chip bonding technique which has allowed for reliable and repeatable manufacture. We have evaluated the performance of the matrix by measuring the different factors including the static, dynamic power consumption, the illumination, and the current consumption by each LED. We show that the power consumption is within a range suitable for portable use. Finally, the thermal behavior of the matrix is monitored and the matrix proved to be thermally stable.

Comparing object recognition from binary and bipolar edge images for visual prostheses

Visual prostheses require an effective representation method due to the limited display condition which has only 2 or 3 levels of grayscale in low resolution. Edges derived from abrupt luminance changes in images carry essential information for object recognition. Typical binary (black and white) edge images have been used to represent features to convey essential information. However, in scenes with a complex cluttered background, the recognition rate of the binary edge images by human observers is limited and additional information is required. The polarity of edges and cusps (black or white features on a gray background) carries important additional information; the polarity may provide shape from shading information missing in the binary edge image. This depth information may be restored by using bipolar edges. We compared object recognition rates from 16 binary edge images and bipolar edge images by 26 subjects to determine the possible impact of bipolar filtering in visual prostheses with 3 or more levels of grayscale. Recognition rates were higher with bipolar edge images and the improvement was significant in scenes with complex backgrounds. The results also suggest that erroneous shape from shading interpretation of bipolar edges resulting from pigment rather than boundaries of shape may confound the recognition.

Blockade of pathological retinal ganglion cell hyperactivity improves optogenetically evoked light responses in rd1 mice

Retinitis pigmentosa (RP) is a progressive retinal dystrophy that causes visual impairment and eventual blindness. Retinal prostheses are the best currently available vision-restoring treatment for RP, but only restore crude vision. One possible contributing factor to the poor quality of vision achieved with prosthetic devices is the pathological retinal ganglion cell (RGC) hyperactivity that occurs in photoreceptor dystrophic disorders. Gap junction blockade with meclofenamic acid (MFA) was recently shown to diminish RGC hyperactivity and improve the signal-to-noise ratio (SNR) of RGC responses to light flashes and electrical stimulation in the rd10 mouse model of RP. We sought to extend these results to spatiotemporally patterned optogenetic stimulation in the faster-degenerating rd1 model and compare the effectiveness of a number of drugs known to disrupt rd1 hyperactivity. We crossed rd1 mice with a transgenic mouse line expressing the light-sensitive cation channel channelrhodopsin2 (ChR2) in RGCs, allowing them to be stimulated directly using high-intensity blue light. We used 60-channel ITO multielectrode arrays to record ChR2-mediated RGC responses from wholemount, ex-vivo retinas to full-field and patterned stimuli before and after application of MFA, 18-β-glycyrrhetinic acid (18BGA, another gap junction blocker) or flupirtine (Flu, a Kv7 potassium channel opener). All three drugs decreased spontaneous RGC firing, but 18BGA and Flu also decreased the sensitivity of RGCs to optogenetic stimulation. Nevertheless, all three drugs improved the SNR of ChR2-mediated responses. MFA also made it easier to discern motion direction of a moving bar from RGC population responses. Our results support the hypothesis that reduction of pathological RGC spontaneous activity characteristic in retinal degenerative disorders may improve the quality of visual responses in retinal prostheses and they provide insights into how best to achieve this for optogenetic prostheses.

Wireless data and power transfer of an optogenetic implantable visual cortex stimulator

In this paper, the wireless data and power transfer for a novel optogenetic visual cortex implant system was demonstrated by using pork tissue mimic in-vitro at the ISM 2.4 GHz and 13.5 MHz frequency band respectively. The observed data rate was 120 kbps with no loss in data for up to a thickness of 35 mm in both water & pork. To increase the power level of the implant a Class E power amplifier is separately designed and simulated for the transmitter end and has an output power of around 223 mW with an efficiency of 81.83%. The transferred power at the receiver was measured to be 66.80 mW for the pork tissue medium considering a distance of 5 mm between the transmitter and the receiver coils, with a coupling coefficient of ~0.8. This serves the power requirement of the visual cortex implant.

Active confocal imaging for visual prostheses

There are encouraging advances in prosthetic vision for the blind, including retinal and cortical implants, and other "sensory substitution devices" that use tactile or electrical stimulation. However, they all have low resolution, limited visual field, and can display only few gray levels (limited dynamic range), severely restricting their utility. To overcome these limitations, image processing or the imaging system could emphasize objects of interest and suppress the background clutter. We propose an active confocal imaging system based on light-field technology that will enable a blind user of any visual prosthesis to efficiently scan, focus on, and "see" only an object of interest while suppressing interference from background clutter. The system captures three-dimensional scene information using a light-field sensor and displays only an in-focused plane with objects in it. After capturing a confocal image, a de-cluttering process removes the clutter based on blur difference. In preliminary experiments we verified the positive impact of confocal-based background clutter removal on recognition of objects in low resolution and limited dynamic range simulated phosphene images. Using a custom-made multiple-camera system based on light-field imaging, we confirmed that the concept of a confocal de-cluttered image can be realized effectively.

Optogenetic approaches to retinal prosthesis

The concept of visual restoration via retinal prosthesis arguably started in 1992 with the discovery that some of the retinal cells were still intact in those with the retinitis pigmentosa disease. Two decades later, the first commercially available devices have the capability to allow users to identify basic shapes. Such devices are still very far from returning vision beyond the legal blindness. Thus, there is considerable continued development of electrode materials, and structures and electronic control mechanisms to increase both resolution and contrast. In parallel, the field of optogenetics--the genetic photosensitization of neural tissue holds particular promise for new approaches. Given that the eye is transparent, photosensitizing remaining neural layers of the eye and illuminating from the outside could prove to be less invasive, cheaper, and more effective than present approaches. As we move toward human trials in the coming years, this review explores the core technological and biological challenges related to the gene therapy and the high radiance optical stimulation requirement.