Simulated disparity and peripheral blur interact during binocular fusion

Low-Latency Ocular Parallax Rendering and Investigation of Its Effect on Depth Perception in Virtual Reality

With a demand for an immersive experience in virtual/augmented reality (VR/AR) displays, recent efforts have incorporated eye states, such as focus and fixation, into display graphics. Among these, ocular parallax, a small parallax generated by eye rotation, has received considerable attention for its impact on depth perception. However, the substantial latency of head-mounted displays (HMDs) has made it challenging to accurately assess its true effect during free eye movements. To address this issue, we propose a high-speed (360 Hz) and low-latency (4.8 ms) ocular parallax rendering system with a custom-built eye tracker. Using this proposed system, we conducted an investigation to determine the latency requirements necessary for achieving perceptually stable ocular parallax rendering. Our findings indicate that, in binocular viewing, ocular parallax rendering is perceived as significantly less stable than conventional rendering when the latency exceeds 43.72 ms at 1.3 D and 21.50 ms at 2.0 D. We also evaluated the effects of ocular parallax rendering on binocular fusion and monocular depth perception under free viewing conditions. The results demonstrated that ocular parallax rendering can enhance binocular fusion but has a limited impact on depth perception under monocular viewing conditions when latency is minimized.

Impact of focus cue presentation on perceived realism of 3-D scene structure: Implications for scene perception and for display technology

Stereoscopic imagery often aims to evoke three-dimensional (3-D) percepts that are accurate and realistic-looking. The "gap" between 3-D imagery and real scenes is small, but focus cues typically remain incorrect because images are displayed on a single focal plane. Research has concentrated on the resulting vergence-accommodation conflicts. Yet, incorrect focus cues may also affect the appearance of 3-D imagery. We investigated whether incorrect focus cues reduce perceived realism of 3-D structure ("depth realism"). Experiment 1 used a multiple-focal-planes display to compare depth realism with correct focus cues vs. conventional stereo presentation. The stimuli were random-dot stereograms, which isolated the role of focus cues. Depth realism was consistently lower with incorrect focus cues, providing proof-of-principle evidence that they contribute to perceptual realism. Experiments 2 and 3 examined whether focus cues play a similar role with realistic objects, presented with an almost complete set of visual cues using a high-resolution, high-dynamic-range multiple-focal-planes display. We also examined the efficacy of approximating correct focus cues via gaze-contingent depth-of-field rendering. Improvements in depth realism with correct focus cues were less clear in more realistic scenes, indicating that the role of focus cues in depth realism depends on scene content. Rendering-based approaches, if anything, reduced depth realism, which we attribute to their inability to present higher-order aspects of blur correctly. Our findings suggest future general 3-D display solutions may need to present focus cues correctly to maximise perceptual realism.

Success rates, near-response patterns, and learning trends with free-fusion stereograms

Free-fusion stereograms are routinely used for demonstrating various stereoscopic effects. Yet, untrained observers find it challenging to perform this task. This study showed that only less than 1/3rd of sixty-one pre-presbyopic adults with normal binocular vision could successfully free-fuse random-dot image pairs and identify the stereoscopic shapes embedded in these patterns. Another one-third of participants performed the task with poor success rates, while the remaining could not perform the task. There was a clear dissociation of vergence and accommodative responses in participants who were successful with free-fusion, as recorded using a dynamic infrared eye tracker and photorefractor. Those in the unsuccessful cluster either showed strong vergence and accommodation or weak vergence and strong accommodation during the task. These response patterns, however, were specific to the free-fusion task because all these participants generated good convergence/accommodation to real-world targets and to conflicting vergence and accommodative demands stimulated with prisms or lenses. Task performance of the unsuccessful cluster also improved significantly following pharmacological paralysis of accommodation and reached the performance levels of the successful cluster. A minority of participants also appeared to progressively learn to dissociate one of the two directions of their vergence and accommodation crosslinks with repeated free-fusion trials. These results suggest that successful free-fusion might depend upon how well participants generate a combination of volitional and reflex vergence responses to large differences in disparity with conflicting static accommodative demands. Such responses would require that only one direction of the vergence-accommodation crosslinks be active at any given time. The sequence of near-responses could also be learnt through repeated trials to optimize task performance.

Blurring the boundary between models and reality: Visual perception of scale assessed by performance

One of the primary jobs of visual perception is to build a three-dimensional representation of the world around us from our flat retinal images. These are a rich source of depth cues but no single one of them can tell us about scale (i.e., absolute depth and size). For example, the pictorial depth cues in a (perfect) scale model are identical to those in the real scene that is being modelled. Here we investigate image blur gradients, which derive naturally from the limited depth of field available for any optical device and can be used to help estimate visual scale. By manipulating image blur artificially to produce what is sometimes called fake tilt shift miniaturization, we provide the first performance-based evidence that human vision uses this cue when making forced-choice judgements about scale (identifying which of an image pair was a photograph of a full-scale railway scene, and which was a 1:76 scale model). The orientation of the blur gradient (relative to the ground plane) proves to be crucial, though its rate of change is less important for our task, suggesting a fairly coarse visual analysis of this image parameter.

Is Peripheral Motion Detection Affected by Myopia?

Purpose

The current study was to investigate whether myopia affected peripheral motion detection and whether the potential effect interacted with spatial frequency, motion speed, or eccentricity.

Methods

Seventeen young adults aged 22–26 years participated in the study. They were six low to medium myopes [spherical equivalent refractions −1.0 to −5.0 D (diopter)], five high myopes (<-5.5 D) and six emmetropes (+0.5 to −0.5 D). All myopes were corrected by self-prepared, habitual soft contact lenses. A four-alternative forced-choice task in which the subject was to determine the location of the phase-shifting Gabor from the four quadrants (superior, inferior, nasal, and temporal) of the visual field, was employed. The experiment was blocked by eccentricity (20° and 27°), spatial frequency (0.6, 1.2, 2.4, and 4.0 cycles per degree (c/d) for 20° eccentricity, and 0.6, 1.2, 2.0, and 3.2 c/d for 27° eccentricity), as well as the motion speed [2 and 6 degree per second (d/s)].

Results

Mixed-model analysis of variances showed no significant difference in the thresholds of peripheral motion detection between three refractive groups at either 20° (F[2,14] = 0.145, p = 0.866) or 27° (F[2,14] = 0.475, p = 0.632). At 20°, lower motion detection thresholds were associated with higher myopia (p < 0.05) mostly for low spatial frequency and high-speed targets in the nasal and superior quadrants, and for high spatial frequency and high-speed targets in the temporal quadrant in myopic viewers. Whereas at 27°, no significant correlation was found between the spherical equivalent and the peripheral motion detection threshold under all conditions (all p > 0.1). Spatial frequency, speed, and quadrant of the visual field all showed significant effect on the peripheral motion detection threshold.

Conclusion

There was no significant difference between the three refractive groups in peripheral motion detection. However, lower motion detection thresholds were associated with higher myopia, mostly for low spatial frequency targets, at 20° in myopic viewers.

A dichoptic feedback-based oculomotor training method to manipulate interocular alignment

Strabismus is a prevalent impairment of binocular alignment that is associated with a spectrum of perceptual deficits and social disadvantages. Current treatments for strabismus involve ocular alignment through surgical or optical methods and may include vision therapy exercises. In the present study, we explore the potential of real-time dichoptic visual feedback that may be used to quantify and manipulate interocular alignment. A gaze-contingent ring was presented independently to each eye of 11 normally-sighted observers as they fixated a target dot presented only to their dominant eye. Their task was to center the rings within 2° of the target for at least 1 s, with feedback provided by the sizes of the rings. By offsetting the ring in the non-dominant eye temporally or nasally, this task required convergence or divergence, respectively, of the non-dominant eye. Eight of 11 observers attained 5° asymmetric convergence and 3 of 11 attained 3° asymmetric divergence. The results suggest that real-time gaze-contingent feedback may be used to quantify and transiently simulate strabismus and holds promise as a method to augment existing therapies for oculomotor alignment disorders.

Near-optimal combination of disparity across a log-polar scaled visual field

The human visual system is foveated: we can see fine spatial details in central vision, whereas resolution is poor in our peripheral visual field, and this loss of resolution follows an approximately logarithmic decrease. Additionally, our brain organizes visual input in polar coordinates. Therefore, the image projection occurring between retina and primary visual cortex can be mathematically described by the log-polar transform. Here, we test and model how this space-variant visual processing affects how we process binocular disparity, a key component of human depth perception. We observe that the fovea preferentially processes disparities at fine spatial scales, whereas the visual periphery is tuned for coarse spatial scales, in line with the naturally occurring distributions of depths and disparities in the real-world. We further show that the visual system integrates disparity information across the visual field, in a near-optimal fashion. We develop a foveated, log-polar model that mimics the processing of depth information in primary visual cortex and that can process disparity directly in the cortical domain representation. This model takes real images as input and recreates the observed topography of human disparity sensitivity. Our findings support the notion that our foveated, binocular visual system has been moulded by the statistics of our visual environment.

We investigate how humans perceive depth from binocular disparity at different spatial scales and across different regions of the visual field. We show that small changes in disparity-defined depth are detected best in central vision, whereas peripheral vision best captures the coarser structure of the environment. We also demonstrate that depth information extracted from different regions of the visual field is combined into a unified depth percept. We then construct an image-computable model of disparity processing that takes into account how our brain organizes the visual input at our retinae. The model operates directly in cortical image space, and neatly accounts for human depth perception across the visual field.

Digitalization versus immersion: performance and subjective evaluation of 3D perception with emulated accommodation and parallax in digital microsurgery

In the visually challenging situation of microsurgery with many altered depth cues, digitalization of surgical systems disrupts two further depth cues, namely focus and parallax. Although in purely optical surgical systems accommodation and eye movements induce expected focus and parallax changes, they become statically fixed through digitalization. Our study evaluates the impact of static focus and parallax onto performance and subjective 3D perception. Subjects reported decreased depth realism under static parallax and focus. Thus surgeons’ depth perception is impacted further through digitalization of microsurgery, increasing the potential of artificial stereo-induced fatigue.

Creating correct blur and its effect on accommodation

Blur occurs naturally when the eye is focused at one distance and an object is presented at another distance. Computer-graphics engineers and vision scientists often wish to create display images that reproduce such depth-dependent blur, but their methods are incorrect for that purpose. They take into account the scene geometry, pupil size, and focal distances, but do not properly take into account the optical aberrations of the human eye. We developed a method that, by incorporating the viewer's optics, yields displayed images that produce retinal images close to the ones that occur in natural viewing. We concentrated on the effects of defocus, chromatic aberration, astigmatism, and spherical aberration and evaluated their effectiveness by conducting experiments in which we attempted to drive the eye's focusing response (accommodation) through the rendering of these aberrations. We found that accommodation is not driven at all by conventional rendering methods, but that it is driven surprisingly quickly and accurately by our method with defocus and chromatic aberration incorporated. We found some effect of astigmatism but none of spherical aberration. We discuss how the rendering approach can be used in vision science experiments and in the development of ophthalmic/optometric devices and augmented- and virtual-reality displays.

Depth variation and stereo processing tasks in natural scenes

Local depth variation is a distinctive property of natural scenes, but its effects on perception have only recently begun to be investigated. Depth variation in natural scenes is due to depth edges between objects and surface nonuniformities within objects. Here, we demonstrate how natural depth variation impacts performance in two fundamental tasks related to stereopsis: half-occlusion detection and disparity detection. We report the results of a computational study that uses a large database of natural stereo-images and coregistered laser-based distance measurements. First, we develop a procedure for precisely sampling stereo-image patches from the stereo-images and then quantify the local depth variation in each patch by its disparity contrast. Next, we show that increased disparity contrast degrades half-occlusion detection and disparity detection performance and changes the size and shape of the spatial integration areas ("receptive fields") that optimize performance. Then, we show that a simple image-computable binocular statistic predicts disparity contrast in natural scenes. Finally, we report the most likely spatial patterns of disparity variation and disparity discontinuities (half-occlusions) in natural scenes. Our findings motivate computational and psychophysical investigations of the mechanisms that underlie stereo processing tasks in local regions of natural scenes.

Three-dimensional binocular eye-hand coordination in normal vision and with simulated visual impairment

Sensorimotor coupling in healthy humans is demonstrated by the higher accuracy of visually tracking intrinsically-rather than extrinsically-generated hand movements in the fronto-parallel plane. It is unknown whether this coupling also facilitates vergence eye movements for tracking objects in depth, or can overcome symmetric or asymmetric binocular visual impairments. Human observers were therefore asked to track with their gaze a target moving horizontally or in depth. The movement of the target was either directly controlled by the observer's hand or followed hand movements executed by the observer in a previous trial. Visual impairments were simulated by blurring stimuli independently in each eye. Accuracy was higher for self-generated movements in all conditions, demonstrating that motor signals are employed by the oculomotor system to improve the accuracy of vergence as well as horizontal eye movements. Asymmetric monocular blur affected horizontal tracking less than symmetric binocular blur, but impaired tracking in depth as much as binocular blur. There was a critical blur level up to which pursuit and vergence eye movements maintained tracking accuracy independent of blur level. Hand-eye coordination may therefore help compensate for functional deficits associated with eye disease and may be employed to augment visual impairment rehabilitation.

Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research

The Tobii Eyex Controller is a new low-cost binocular eye tracker marketed for integration in gaming and consumer applications. The manufacturers claim that the system was conceived for natural eye gaze interaction, does not require continuous recalibration, and allows moderate head movements. The Controller is provided with a SDK to foster the development of new eye tracking applications. We review the characteristics of the device for its possible use in scientific research. We develop and evaluate an open source Matlab Toolkit that can be employed to interface with the EyeX device for gaze recording in behavioral experiments. The Toolkit provides calibration procedures tailored to both binocular and monocular experiments, as well as procedures to evaluate other eye tracking devices. The observed performance of the EyeX (i.e. accuracy < 0.6°, precision < 0.25°, latency < 50 ms and sampling frequency ≈55 Hz), is sufficient for some classes of research application. The device can be successfully employed to measure fixation parameters, saccadic, smooth pursuit and vergence eye movements. However, the relatively low sampling rate and moderate precision limit the suitability of the EyeX for monitoring micro-saccadic eye movements or for real-time gaze-contingent stimulus control. For these applications, research grade, high-cost eye tracking technology may still be necessary. Therefore, despite its limitations with respect to high-end devices, the EyeX has the potential to further the dissemination of eye tracking technology to a broad audience, and could be a valuable asset in consumer and gaming applications as well as a subset of basic and clinical research settings.

Defocus Discrimination in Video: Motion in Depth

We perform two psychophysics experiments to investigate a viewer's ability to detect defocus in video; in particular, the defocus that arises in video during motion in depth when the camera does not maintain sharp focus throughout the motion. The first experiment demonstrates that blur sensitivity during viewing is affected by the speed at which the target moves towards the camera. The second experiment measures a viewer's ability to notice momentary defocus and shows that the threshold of blur detection in arc minutes decreases significantly as the duration of the blur increases. Our results suggest that it is important to have good control of focus while recording video and that momentary defocus should be kept as short as possible so it goes unnoticed.

Vergence driven accommodation with simulated disparity in myopia and emmetropia

The formation of focused and corresponding foveal images requires a close synergy between the accommodation and vergence systems. This linkage is usually decoupled in virtual reality systems and may be dysfunctional in people who are at risk of developing myopia. We study how refractive error affects vergence-accommodation interactions in stereoscopic displays. Vergence and accommodative responses were measured in 21 young healthy adults (n=9 myopes, 22-31 years) while subjects viewed naturalistic stimuli on a 3D display. In Step 1, vergence was driven behind the monitor using a blurred, non-accommodative, uncrossed disparity target. In Step 2, vergence and accommodation were driven back to the monitor plane using naturalistic images that contained structured depth and focus information from size, blur and/or disparity. In Step 1, both refractive groups converged towards the stereoscopic target depth plane, but the vergence-driven accommodative change was smaller in emmetropes than in myopes (F_1,19=5.13, p=0.036). In Step 2, there was little effect of peripheral depth cues on accommodation or vergence in either refractive group. However, vergence responses were significantly slower (F_1,19=4.55, p=0.046) and accommodation variability was higher (F_1,19=12.9, p=0.0019) in myopes. Vergence and accommodation responses are disrupted in virtual reality displays in both refractive groups. Accommodation responses are less stable in myopes, perhaps due to a lower sensitivity to dioptric blur. Such inaccuracies of accommodation may cause long-term blur on the retina, which has been associated with a failure of emmetropization.

How to Build a Dichoptic Presentation System That Includes an Eye Tracker

The presentation of different stimuli to the two eyes, dichoptic presentation, is essential for studies involving 3D vision and interocular suppression. There is a growing literature on the unique experimental value of pupillary and oculomotor measures, especially for research on interocular suppression. Although obtaining eye-tracking measures would thus benefit studies that use dichoptic presentation, the hardware essential for dichoptic presentation (e.g. mirrors) often interferes with high-quality eye tracking, especially when using a video-based eye tracker. We recently described an experimental setup that combines a standard dichoptic presentation system with an infrared eye tracker by using infrared-transparent mirrors1. The setup is compatible with standard monitors and eye trackers, easy to implement, and affordable (on the order of US$1,000). Relative to existing methods it has the benefits of not requiring special equipment and posing few limits on the nature and quality of the visual stimulus. Here we provide a visual guide to the construction and use of our setup.

Blur perception throughout the visual field in myopia and emmetropia

We evaluated the ability of emmetropic and myopic observers to detect and discriminate blur across the retina under monocular or binocular viewing conditions. We recruited 39 young (23-30 years) healthy adults (n = 19 myopes) with best-corrected visual acuity 0.0 LogMAR (20/20) or better in each eye and no binocular or accommodative dysfunction. Monocular and binocular blur discrimination thresholds were measured as a function of pedestal blur using naturalistic stimuli with an adaptive 4AFC procedure. Stimuli were presented in a 46° diameter window at 40 cm. Gaussian blur pedestals were confined to an annulus at either 0°, 4°, 8°, or 12° eccentricity, with a blur increment applied to only one quadrant of the image. The adaptive procedure efficiently estimated a dipper shaped blur discrimination threshold function with two parameters: intrinsic blur and blur sensitivity. The amount of intrinsic blur increased for retinal eccentricities beyond 4° (p < 0.001) and was lower in binocular than monocular conditions (p < 0.001), but was similar across refractive groups (p = 0.47). Blur sensitivity decreased with retinal eccentricity (p < 0.001) and was highest for binocular viewing, but only for central vision (p < 0.05). Myopes showed worse blur sensitivity than emmetropes monocularly (p < 0.05) but not binocularly (p = 0.66). As expected, blur perception worsens in the visual periphery and binocular summation is most evident in central vision. Furthermore, myopes exhibit a monocular impairment in blur sensitivity that improves under binocular conditions. Implications for the development of myopia are discussed.

3D Displays

Creating realistic three-dimensional (3D) experiences has been a very active area of research and development, and this article describes progress and what remains to be solved. A very active area of technical development has been to build displays that create the correct relationship between viewing parameters and triangulation depth cues: stereo, motion, and focus. Several disciplines are involved in the design, construction, evaluation, and use of 3D displays, but an understanding of human vision is crucial to this enterprise because in the end, the goal is to provide the desired perceptual experience for the viewer. In this article, we review research and development concerning displays that create 3D experiences. And we highlight areas in which further research and development is needed.

Dynamic lens and monovision 3D displays to improve viewer comfort

Stereoscopic 3D (S3D) displays provide an additional sense of depth compared to non-stereoscopic displays by sending slightly different images to the two eyes. But conventional S3D displays do not reproduce all natural depth cues. In particular, focus cues are incorrect causing mismatches between accommodation and vergence: The eyes must accommodate to the display screen to create sharp retinal images even when binocular disparity drives the eyes to converge to other distances. This mismatch causes visual discomfort and reduces visual performance. We propose and assess two new techniques that are designed to reduce the vergence-accommodation conflict and thereby decrease discomfort and increase visual performance. These techniques are much simpler to implement than previous conflict-reducing techniques. The first proposed technique uses variable-focus lenses between the display and the viewer's eyes. The power of the lenses is yoked to the expected vergence distance thereby reducing the mismatch between vergence and accommodation. The second proposed technique uses a fixed lens in front of one eye and relies on the binocularly fused percept being determined by one eye and then the other, depending on simulated distance. We conducted performance tests and discomfort assessments with both techniques and compared the results to those of a conventional S3D display. The first proposed technique, but not the second, yielded clear improvements in performance and reductions in discomfort. This dynamic-lens technique therefore offers an easily implemented technique for reducing the vergence-accommodation conflict and thereby improving viewer experience.

The (In)Effectiveness of Simulated Blur for Depth Perception in Naturalistic Images

We examine depth perception in images of real scenes with naturalistic variation in pictorial depth cues, simulated dioptric blur and binocular disparity. Light field photographs of natural scenes were taken with a Lytro plenoptic camera that simultaneously captures images at up to 12 focal planes. When accommodation at any given plane was simulated, the corresponding defocus blur at other depth planes was extracted from the stack of focal plane images. Depth information from pictorial cues, relative blur and stereoscopic disparity was separately introduced into the images. In 2AFC tasks, observers were required to indicate which of two patches extracted from these images was farther. Depth discrimination sensitivity was highest when geometric and stereoscopic disparity cues were both present. Blur cues impaired sensitivity by reducing the contrast of geometric information at high spatial frequencies. While simulated generic blur may not assist depth perception, it remains possible that dioptric blur from the optics of an observer’s own eyes may be used to recover depth information on an individual basis. The implications of our findings for virtual reality rendering technology are discussed.

On the number of perceivable blur levels in naturalistic images

Blur is a useful cue for depth. Natural images contain objects at a range of depths whose depth can be signaled by their perceived blur. Here, to evaluate the usefulness of blur as a depth cue, we estimate the number blur levels that observers can perceive simultaneously. To estimate this value, observers discriminated and classified dead leaves patterns that contained a controlled distribution of blur levels but are more complex or naturalistic than stimuli typically used in blur research. We used a 2-IFC discrimination task, in which observers reported the interval that contained more blur levels and a classification task, in which observers reported the number of perceived blur levels. In both tasks, observers could not discriminate or classify more than four levels of blur in the stimulus reliably. In isolation from other cues, blur may provide only a coarse cue to depth and add limited depth information when present in natural scenes with complex distributions of blur and multiple depth cues.