Performance of complex visual tasks using simulated prosthetic vision via augmented-reality glasses

Precise control of neural activity using dynamically optimized electrical stimulation

Neural implants have the potential to restore lost sensory function by electrically evoking the complex naturalistic activity patterns of neural populations. However, it can be difficult to predict and control evoked neural responses to simultaneous multi-electrode stimulation due to nonlinearity of the responses. We present a solution to this problem and demonstrate its utility in the context of a bidirectional retinal implant for restoring vision. A dynamically optimized stimulation approach encodes incoming visual stimuli into a rapid, greedily chosen, temporally dithered and spatially multiplexed sequence of simple stimulation patterns. Stimuli are selected to optimize the reconstruction of the visual stimulus from the evoked responses. Temporal dithering exploits the slow time scales of downstream neural processing, and spatial multiplexing exploits the independence of responses generated by distant electrodes. The approach was evaluated using an experimental laboratory prototype of a retinal implant: large-scale, high-resolution multi-electrode stimulation and recording of macaque and rat retinal ganglion cells ex vivo. The dynamically optimized stimulation approach substantially enhanced performance compared to existing approaches based on static mapping between visual stimulus intensity and current amplitude. The modular framework enabled parallel extensions to naturalistic viewing conditions, incorporation of perceptual similarity measures, and efficient implementation for an implantable device. A direct closed-loop test of the approach supported its potential use in vision restoration.

Simulation of visual impairment in persons with normal vision for scientific research

Simulation of visual impairment in healthy eyes has multiple applications in students' training, research and product development. However, due to the absence of an existing standard protocol, the method of simulation was left to the discretion of the researcher. This review aimed to outline the various methods of simulating visual impairment and categorising them. A scoping review of the relevant publications was conducted. Of the 1593 articles originally retrieved from the databases, 103 were included in the review. The characteristics of the participants, the method for simulation of the visual impairment in persons with normal vision and the level or type of visual impairment that was simulated were extracted from the papers. None of the methods of simulation can be judged as being superior to the others. However, electronic displays produced the most consistent form of visual impairment simulation.

A novel simulation paradigm utilising MRI-derived phosphene maps for cortical prosthetic vision

Objective.We developed a realistic simulation paradigm for cortical prosthetic vision and investigated whether we can improve visual performance using a novel clustering algorithm.Approach.Cortical visual prostheses have been developed to restore sight by stimulating the visual cortex. To investigate the visual experience, previous studies have used uniform phosphene maps, which may not accurately capture generated phosphene map distributions of implant recipients. The current simulation paradigm was based on the Human Connectome Project retinotopy dataset and the placement of implants on the cortices from magnetic resonance imaging scans. Five unique retinotopic maps were derived using this method. To improve performance on these retinotopic maps, we enabled head scanning and a density-based clustering algorithm was then used to relocate centroids of visual stimuli. The impact of these improvements on visual detection performance was tested. Using spatially evenly distributed maps as a control, we recruited ten subjects and evaluated their performance across five sessions on the Berkeley Rudimentary Visual Acuity test and the object recognition task.Main results.Performance on control maps is significantly better than on retinotopic maps in both tasks. Both head scanning and the clustering algorithm showed the potential of improving visual ability across multiple sessions in the object recognition task.Significance.The current paradigm is the first that simulates the experience of cortical prosthetic vision based on brain scans and implant placement, which captures the spatial distribution of phosphenes more realistically. Utilisation of evenly distributed maps may overestimate the performance that visual prosthetics can restore. This simulation paradigm could be used in clinical practice when making plans for where best to implant cortical visual prostheses.

A systematic review of extended reality (XR) for understanding and augmenting vision loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user's eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

Grand Theft Auto-Based Cycling Simulator for Cognitive Enhancement Technologies in Dangerous Traffic Situations

While developing traffic-based cognitive enhancement technology (CET), such as bike accident prevention systems, it can be challenging to test and evaluate them properly. After all, the real-world scenario could endanger the subjects' health and safety. Therefore, a simulator is needed, preferably one that is realistic yet low cost. This paper introduces a way to use the video game Grand Theft Auto V (GTA V) and its sophisticated traffic system as a base to create such a simulator, allowing for the safe and realistic testing of dangerous traffic situations involving cyclists, cars, and trucks. The open world of GTA V, which can be explored on foot and via various vehicles, serves as an immersive stand-in for the real world. Custom modification scripts of the game give the researchers control over the experiment scenario and the output data to be evaluated. An off-the-shelf bicycle equipped with three sensors serves as a realistic input device for the subject's movement direction and speed. The simulator was used to test two early-stage CET concepts enabling cyclists to sense dangerous traffic situations, such as trucks approaching from behind the cyclist. Thus, this paper also presents the user evaluation of the cycling simulator and the CET used by the subjects to sense dangerous traffic situations. With the knowledge of the first iteration of the user-centered design (UCD) process, this paper concludes by naming improvements for the cycling simulator and discussing further research directions for CET that enable users to sense dangerous situations better.

Measuring visual information gathering in individuals with ultra low vision using virtual reality

People with ULV (visual acuity ≤ 20/1600 or 1.9 logMAR) lack form vision but have rudimentary levels of vision that can be used for a range of activities in daily life. However, current clinical tests are designed to assess form vision and do not provide information about the range of visually guided activities that can be performed in daily life using ULV. This is important to know given the growing number of clinical trials that recruit individuals with ULV (e.g., gene therapy, stem cell therapy) or restore vision to the ULV range in the blind (visual prosthesis). In this study, we develop a set of 19 activities (items) in virtual reality involving spatial localization/detection, motion detection, and direction of motion that can be used to assess visual performance in people with ULV. We estimated measures of item difficulty and person ability on a relative d prime (d') axis using a signal detection theory based analysis for latent variables. The items represented a range of difficulty levels (- 1.09 to 0.39 in relative d') in a heterogeneous group of individuals with ULV (- 0.74 to 2.2 in relative d') showing the instrument's utility as an outcome measure in clinical trials.

The Study of Short-Term Plastic Visual Perceptual Training Based on Virtual and Augmented Reality Technology in Amblyopia

Backgrounds. The treatment for amblyopia can have a substantial impact on quality of life. Conventional treatments for amblyopia have some limitations, then we try to explore a new and effective method to treat amblyopia. This study aimed to determine the potential effect of short-term plastic visual perceptual training based on VR and AR platforms in amblyopic patients. Methods. All observers were blinded to patient groupings. A total of 145 amblyopic children were randomly assigned into 2 groups: VR group (71 patients) and AR group (74 patients). In the VR group, each subject underwent a 20-min short-term plastic visual perceptual training based on a VR platform, and in the AR group, based on an AR platform. The best-corrected visual acuity (BCVA), fine stereopsis, and contrast sensitivity function (CSF) were measured before and after training. Results. The BCVA (P < 0.001) and fine stereopsis (P < 0.05) were improved significantly both in VR and AR group after training. Moreover, in the AR group, the CSF showed the value of all spatial frequencies had a statistically significant improvement after training (P < 0.05), while in the VR group, only the value of spatial frequency 12 improved significantly (P = 0.008). Conclusions. This study showed that the short-term plastic visual perceptual training based on VR and AR technology can improve BCVA, fine stereopsis and CSF of refractive amblyopia. It was suggested that the visual perceptual training based on the VR and AR platforms may be potentially applied in treatment for amblyopia and provided a high-immersing alternative.

Virtual reality validation of naturalistic modulation strategies to counteract fading in retinal stimulation

OBJECTIVE

Temporal resolution is a key challenge in artificial vision. Several prosthetic approaches are limited by the perceptual fading of evoked phosphenes upon repeated stimulation from the same electrode. Therefore, implanted patients are forced to perform active scanning, via head movements, to refresh the visual field viewed by the camera. However, active scanning is a draining task, and it is crucial to find compensatory strategies to reduce it.

APPROACH

To address this question, we implemented perceptual fading in simulated prosthetic vision using virtual reality. Then, we quantified the effect of fading on two indicators: the time to complete a reading task and the head rotation during the task. We also tested if stimulation strategies previously proposed to increase the persistence of responses in retinal ganglion cells to electrical stimulation could improve these indicators.

MAIN RESULTS

This study shows that stimulation strategies based on interrupted pulse trains and randomisation of the pulse duration allows significant reduction of both the time to complete the task and the head rotation during the task.

SIGNIFICANCE

The stimulation strategy used in retinal implants is crucial to counteract perceptual fading and to reduce active head scanning during prosthetic vision. In turn, less active scanning might improve the patient's comfort in artificial vision.

Simulating Macular Degeneration to Investigate Activities of Daily Living: A Systematic Review

Purpose: Investigating difficulties during activities of daily living is a fundamental first step for the development of vision-related intervention and rehabilitation strategies. One way to do this is through visual impairment simulations. The aim of this review is to synthesize and assess the types of simulation methods that have been used to simulate age-related macular degeneration (AMD) in normally sighted participants, during activities of daily living (e.g., reading, cleaning, and cooking).

Methods: We conducted a systematic literature search in five databases and a critical analysis of the advantages and disadvantages of various AMD simulation methods (following PRISMA guidelines). The review focuses on the suitability of each method for investigating activities of daily living, an assessment of clinical validation procedures, and an evaluation of the adaptation periods for participants.

Results: Nineteen studies met the criteria for inclusion. Contact lenses, computer manipulations, gaze contingent displays, and simulation glasses were the main forms of AMD simulation identified. The use of validation and adaptation procedures were reported in approximately two-thirds and half of studies, respectively.

Conclusions: Synthesis of the methodology demonstrated that the choice of simulation has been, and should continue to be, guided by the nature of the study. While simulations may never completely replicate vision loss experienced during AMD, consistency in simulation methodology is critical for generating realistic behavioral responses under vision impairment simulation and limiting the influence of confounding factors. Researchers could also come to a consensus regarding the length and form of adaptation by exploring what is an adequate amount of time and type of training required to acclimatize participants to vision impairment simulations.

Emergence of non-artificial intelligence digital health innovations in ophthalmology: A systematic review

The prominent rise of digital health in ophthalmology is evident in the current age of Industry 4.0. Despite the many facets of digital health, there has been a greater slant in interest and focus on artificial intelligence recently. Other major elements of digital health like wearables could also substantially impact patient-focused outcomes but have been relatively less explored and discussed. In this review, we comprehensively evaluate the use of non-artificial intelligence digital health tools in ophthalmology. 53 papers were included in this systematic review - 25 papers discuss virtual or augmented reality, 14 discuss mobile applications and 14 discuss wearables. Most papers focused on the use of technologies to detect or rehabilitate visual impairment, glaucoma and age-related macular degeneration. Overall, the findings on patient-focused outcomes with the adoption of these technologies are encouraging. Further validation, large-scale studies and earlier consideration of real-world barriers are warranted to enable better real-world implementation.

Applications of augmented reality in ophthalmology [Invited]

Throughout the last decade, augmented reality (AR) head-mounted displays (HMDs) have gradually become a substantial part of modern life, with increasing applications ranging from gaming and driver assistance to medical training. Owing to the tremendous progress in miniaturized displays, cameras, and sensors, HMDs are now used for the diagnosis, treatment, and follow-up of several eye diseases. In this review, we discuss the current state-of-the-art as well as potential uses of AR in ophthalmology. This review includes the following topics: (i) underlying optical technologies, displays and trackers, holography, and adaptive optics; (ii) accommodation, 3D vision, and related problems such as presbyopia, amblyopia, strabismus, and refractive errors; (iii) AR technologies in lens and corneal disorders, in particular cataract and keratoconus; (iv) AR technologies in retinal disorders including age-related macular degeneration (AMD), glaucoma, color blindness, and vision simulators developed for other types of low-vision patients.