A systematic review of extended reality (XR) for understanding and augmenting vision loss

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user's eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person's residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

Reading text works better than watching videos to improve acuity in a simulation of artificial vision

Simulated artificial vision is used in visual prosthesis design to answer questions about device usability. We previously reported a striking increase in equivalent visual acuity with daily use of a simulation of artificial vision in an active task, reading sentences, that required high levels of subject engagement, but passive activities are more likely to dominate post-implant experience. Here, we investigated the longitudinal effects of a passive task, watching videos. Eight subjects used a simulation of a thalamic visual prosthesis with 1000 phosphenes to watch 23 episodes of classic American television in daily, 25-min sessions, for a period of 1 month with interspersed reading tests that quantified reading accuracy and reading speed. For reading accuracy, we found similar dynamics to the early part of the learning process in our previous report, here leading to an improvement in visual acuity of 0.15 ± 0.05 logMAR. For reading speed, however, no change was apparent by the end of training. We found that single reading sessions drove about twice the improvement in acuity of single video sessions despite being only half as long. We conclude that while passive viewing tasks may prove useful for post-implant rehabilitation, active tasks are likely to be preferable.

Simulating the perceptual effects of electrode-retina distance in prosthetic vision

OBJECTIVE

Retinal prostheses aim to restore some vision in retinitis pigmentosa and age-related macular degeneration blind patients. Many spatial and temporal aspects have been found to affect prosthetic vision. Our objective is to study the impact of the space-variant distance between the stimulating electrodes and the surface of the retina on prosthetic vision and how to mitigate this impact.

APPROACH

A prosthetic vision simulation was built to demonstrate the perceptual effects of the electrode-retina distance (ERD) with different random spatial variations, such as size, brightness, shape, dropout, and spatial shifts. Three approaches for reducing the ERD effects are demonstrated: electrode grouping (quads), ERD-based input-image enhancement, and object scanning with and without phosphene persistence. A quantitative assessment for the first two approaches was done based on experiments with 20 subjects and three vision-based computational image similarity metrics.

MAIN RESULTS

The effects of various ERDs on phosphenes' size, brightness, and shape were simulated. Quads, chosen according to the ERDs, effectively elicit phosphenes without exceeding the safe charge density limit, whereas single electrodes with large ERD cannot do so. Input-image enhancement reduced the ERD effects effectively. These two approaches significantly improved ERD-affected prosthetic vision according to the experiment and image similarity metrics. A further reduction of the ERD effects was achieved by scanning an object while moving the head.

SIGNIFICANCE

ERD has multiple effects on perception with retinal prostheses. One of them is vision loss caused by the incapability of electrodes with large ERD to evoke phosphenes. The three approaches presented in this study can be used separately or together to mitigate the impact of ERD. A consideration of our approaches in reducing the perceptual effects of the ERD may help improve the perception with current prosthetic technology and influence the design of future prostheses.

Effects of Depth-Based Object Isolation in Simulated Retinal Prosthetic Vision

Visual retinal prostheses aim to restore vision for blind individuals who suffer from outer retinal degenerative diseases, such as retinitis pigmentosa and age-related macular degeneration. Perception through retinal prostheses is very limited, but it can be improved by applying object isolation. We used an object isolation algorithm based on integral imaging to isolate objects of interest according to their depth from the camera and applied image processing manipulation to the isolated-object images. Subsequently, we applied a spatial prosthetic vision simulation that converted the isolated-object images to phosphene images. We compared the phosphene images for two types of input images, the original image (before applying object isolation), and the isolated-object image to illustrate the effects of object isolation on simulated prosthetic vision without and with multiple spatial variations of phosphenes, such as size and shape variations, spatial shifts, and dropout rate. The results show an improvement in the perceived shape, contrast, and dynamic range (number of gray levels) of objects in the phosphene image.

Retinal prosthetic vision simulation: temporal aspects

Objective. The perception of individuals fitted with retinal prostheses is not fully understood, although several retinal implants have been tested and commercialized. Realistic simulations of perception with retinal implants would be useful for future development and evaluation of such systems.Approach.We implemented a retinal prosthetic vision simulation, including temporal features, which have not been previously simulated. In particular, the simulation included temporal aspects such as persistence and perceptual fading of phosphenes and the electrode activation rate.Main results.The simulated phosphene persistence showed an effective reduction in flickering at low electrode activation rates. Although persistence has a positive effect on static scenes, it smears dynamic scenes. Perceptual fading following continuous stimulation affects prosthetic vision of both static and dynamic scenes by making them disappear completely or partially. However, we showed that perceptual fading of a static stimulus might be countered by head-scanning motions, which together with the persistence revealed the contours of the faded object. We also showed that changing the image polarity may improve simulated prosthetic vision in the presence of persistence and perceptual fading.Significance.Temporal aspects have important roles in prosthetic vision, as illustrated by the simulations. Considering these aspects may improve the future design, the training with, and evaluation of retinal prostheses.

Low Vision Enhancement with Head-mounted Video Display Systems: Are We There Yet?

SIGNIFICANCE

Head-mounted video display systems and image processing as a means of enhancing low vision are ideas that have been around for more than 20 years. Recent developments in virtual and augmented reality technology and software have opened up new research opportunities that will lead to benefits for low vision patients. Since the Visionics low vision enhancement system (LVES), the first head-mounted video display LVES, was engineered 20 years ago, various other devices have come and gone with a recent resurgence of the technology over the past few years. In this article, we discuss the history of the development of LVESs, describe the current state of available technology by outlining existing systems, and explore future innovation and research in this area. Although LVESs have now been around for more than two decades, there is still much that remains to be explored. With the growing popularity and availability of virtual reality and augmented reality technologies, we can now integrate these methods within low vision rehabilitation to conduct more research on customized contrast-enhancement strategies, image motion compensation, image-remapping strategies, and binocular disparity, all while incorporating eye-tracking capabilities. Future research should use this available technology and knowledge to learn more about the visual system in the low vision patient and extract this new information to create prescribable vision enhancement solutions for the visually impaired individual.

Improvement in reading performance through training with simulated thalamic visual prostheses

Simulations of artificial vision are used to provide the researcher an opportunity to explore different aspects of visual prosthesis device design by observing subject performance on various tasks viewed through the simulation. Such studies typically use normal, sighted subjects to measure performance at a given point in time. Relatively few studies examine performance changes longitudinally to quantitatively assess the benefits from a training plan that would be akin to post-implantation rehabilitation. Here, we had six normal, sighted subjects use a standard reading task with daily practice over eight weeks to understand the effects of an intensive training schedule on adaptation to artificial sight. Subjects read 40 MNREAD-style sentences per session, with a new set each session, that were presented at five font sizes (logMAR 1.0–1.4) and through three center-weighted phosphene patterns (2,000, 1,000, 500 phosphenes). We found that subjects improved their reading accuracy across sessions, and that the training lead to an increase of reading speed that was equivalent to a doubling of available phosphenes. Most importantly, the hardest condition, while initially illegible, supported functional reading after training. Consistent with experience-driven neuroplastic changes, gaps in the training schedule lead to transient decreases in reading speed, but, surprisingly, not reading accuracy. Our findings contribute to our larger project of developing a thalamic visual prosthesis and to post-implant rehabilitation strategies.

An optimized content-aware image retargeting method: toward expanding the perceived visual field of the high-density retinal prosthesis recipients

OBJECTIVE

Retinal prosthesis devices have shown great value in restoring some sight for individuals with profoundly impaired vision, but the visual acuity and visual field provided by prostheses greatly limit recipients' visual experience. In this paper, we employ computer vision approaches to seek to expand the perceptible visual field in patients implanted potentially with a high-density retinal prosthesis while maintaining visual acuity as much as possible.

APPROACH

We propose an optimized content-aware image retargeting method, by introducing salient object detection based on color and intensity-difference contrast, aiming to remap important information of a scene into a small visual field and preserve their original scale as much as possible. It may improve prosthetic recipients' perceived visual field and aid in performing some visual tasks (e.g. object detection and object recognition). To verify our method, psychophysical experiments, detecting object number and recognizing objects, are conducted under simulated prosthetic vision. As control, we use three other image retargeting techniques, including Cropping, Scaling, and seam-assisted shrinkability.

MAIN RESULTS

Results show that our method outperforms in preserving more key features and has significantly higher recognition accuracy in comparison with other three image retargeting methods under the condition of small visual field and low-resolution.

SIGNIFICANCE

The proposed method is beneficial to expand the perceived visual field of prosthesis recipients and improve their object detection and recognition performance. It suggests that our method may provide an effective option for image processing module in future high-density retinal implants.

Artificial Organs 2015: A Year in Review

In this Editor's Review, articles published in 2015 are organized by category and briefly summarized. We aim to provide a brief reflection of the currently available worldwide knowledge that is intended to advance and better human life while providing insight for continued application of technologies and methods of organ Replacement, Recovery, and Regeneration. As the official journal of The International Federation for Artificial Organs, The International Faculty for Artificial Organs, the International Society for Rotary Blood Pumps, the International Society for Pediatric Mechanical Cardiopulmonary Support, and the Vienna International Workshop on Functional Electrical Stimulation, Artificial Organs continues in the original mission of its founders "to foster communications in the field of artificial organs on an international level." Artificial Organs continues to publish developments and clinical applications of artificial organ technologies in this broad and expanding field of organ Replacement, Recovery, and Regeneration from all over the world. We take this time also to express our gratitude to our authors for providing their work to this journal. We offer our very special thanks to our reviewers who give so generously of their time and expertise to review, critique, and especially provide meaningful suggestions to the author's work whether eventually accepted or rejected. Without these excellent and dedicated reviewers, the quality expected from such a journal could not be possible. We also express our special thanks to our Publisher, John Wiley & Sons for their expert attention and support in the production and marketing of Artificial Organs. We look forward to reporting further advances in the coming years.