Super multi-view near-eye display to solve vergence-accommodation conflict

Chromatic correction for super multi-view 3D display based on diffraction theory

The traditional analysis method for super multi-view 3D display based on geometric optics, which approximates the lenticular lenses as a series of pinhole structures, ignored the chromatic aberration. In this paper, the optimization method based on diffraction theory is proposed for super multi-view 3D display, where the wavefronts are evaluated accurately by the forward propagation method, and the chromatic aberration of the synthetic viewpoint image is reduced dramatically by the backward reconstruction optimization method (BROM). The optical experiment is performed to verify the feasibility of the method, which is consistent with numerical simulation results. It is proved that the proposed method simulates the physical propagation process of super multi-view 3D display and improves the reconstructed image quality. In the future, it can be used to achieve the super multi-view 3D light field technology with low crosstalk.

Enhancing the spatial resolution of light-field displays without losing angular resolution by a computational subpixel realignment

Low spatial resolution is an urgent problem in integral imaging light-field displays (LFDs). This study proposes a computational method to enhance the spatial resolution without losing angular resolution. How rays reconstruct voxels through lenslets is changed so that every ray through a lenslet merely provides a subpixel. The three subpixels of a pixel no longer form one voxel but three independent voxels. We further demonstrate imperfect integration of subpixels, called the sampling error, can be eliminated on specific image depths, including the central depth plane. By realigning subpixels in the above manner under no sampling error, the sampling rate of voxels is three times the conventional pixel-based LFDs. Moreover, the ray number of every voxel is preserved for an unaffected angular resolution. With unavoidable component alignment errors, resolution gains of 2.52 and 2.0 are verified in simulation and experiment by computationally updating the elemental image array. The proposed computational method further reveals that LFDs intrinsically have a higher space-bandwidth product than presumed.

A Depth-Enhanced Holographic Super Multi-View Display Based on Depth Segmentation

A super multi-view (SMV) near-eye display (NED) effectively provides depth cues for three-dimensional (3D) display by projecting multiple viewpoint or parallax images onto the retina simultaneously. Previous SMV NED have suffered from a limited depth of field (DOF) due to a fixed image plane. In this paper, a holographic SMV Maxwellian display based on depth segmentation is proposed to enhance the DOF. The proposed approach involves capturing a set of parallax images and their corresponding depth maps. According to the depth maps, the parallax images are segmented into N sub-parallax images at different depth ranges. These sub-parallax images are then projected onto N image-recording planes (IRPs) of the corresponding depth for hologram computation. The wavefront at each IRP is calculated by multiplying the sub-parallax images with the corresponding spherical wave phases. Then, they are propagated to the hologram plane and added together to form a DOF-enhanced hologram. The simulation and experimental results are obtained to validate the effectiveness of the proposed method in extending the DOF of the holographic SMV displays, while accurately preserving occlusion.

Extended depth of field in augmented reality

The 3D display device shows an image with depth information. Conventional 3D display devices based on binocular parallax can focus accurately only on the depth of a specific screen. Because the human eye has a narrow depth of field (DOF) under normal circumstances, 3D displays that provide a relatively wide range of virtual depth areas have limitations on the DOF where clear 3D images are seen. To resolve this problem, it is necessary to find the optical conditions to extend the DOF and analyze the phenomena related to it. For this, by using the Rayleigh criterion and the Strehl ratio, a criterion for this extension of the DOF is suggested. A practical optical structure that can effectively extend the DOF is devised using a flat panel display. This optical structure could be applied to AR, VR, and MR in the field of near-eye displays. From the results of this research, the fundamental optical conditions and standards are proposed for 3D displays that will provide 3D images with extended DOF in the future. Furthermore, it is also expected that these conditions and criteria can be applied to optical designs for the required performance in the development of 3D displays in various fields.

Depth-Enhanced Holographic Super Multi-View Maxwellian Display Based on Variable Filter Aperture

The super multi-view (SMV) near-eye display (NED) effectively provides depth cues for three-dimensional (3D) displays by projecting multiple viewpoint images or parallax images onto the retina simultaneously. Previous SMV NED suffers from a limited depth of field (DOF) due to the fixed image plane. Aperture filtering is widely used to enhance the DOF; however, an invariably sized aperture may have opposite effects on objects with different reconstruction depths. In this paper, a holographic SMV display based on the variable filter aperture is proposed to enhance the DOF. In parallax image acquisition, multiple groups of parallax images, each group recording a part of the 3D scene on a fixed depth range, are captured first. In the hologram calculation, each group of wavefronts at the image recording plane (IRP) is calculated by multiplying the parallax images with the corresponding spherical wave phase. Then, they are propagated to the pupil plane and multiplied by the corresponding aperture filter function. The size of the filter aperture is variable which is determined by the depth of the object. Finally, the complex amplitudes at the pupil plane are back-propagated to the holographic plane and added together to form the DOF-enhanced hologram. Simulation and experimental results verify the proposed method could improve the DOF of holographic SMV display, which will contribute to the application of 3D NED.

Super multi-view display based on near-eye pinholes

Vergence-accommodation-conflict-free super multi-view (SMV) display based on more than one near-eye pinhole group for each pupil of the viewer is developed in this paper. Two-dimensionally arranged pinholes of a group correspond to different subscreens of the display screen, with perspective views projected by each subscreen through a corresponding pinhole splicing into an image of enlarged field of view (FOV). By sequentially switching on/off different pinhole groups, more than one mosaic image is projected to each pupil of the viewer. Adjacent pinholes of a group are endowed with different timing-polarizing characteristics to generate an effective noise-free region for each pupil. In the experiment, four groups of 3×3 pinholes are configured for a proof-of-concept SMV display on a 240 Hz display screen, with a diagonal FOV of 55 deg and a depth of field reaching 1.2 m.

The Statistics of Eye Movements and Binocular Disparities during VR Gaming: Implications for Headset Design

The human visual system evolved in environments with statistical regularities. Binocular vision is adapted to these such that depth perception and eye movements are more precise, faster, and performed comfortably in environments consistent with the regularities. We measured the statistics of eye movements and binocular disparities in virtual-reality (VR) - gaming environments and found that they are quite different from those in the natural environment. Fixation distance and direction are more restricted in VR, and fixation distance is farther. The pattern of disparity across the visual field is less regular in VR and does not conform to a prominent property of naturally occurring disparities. From this we predict that double vision is more likely in VR than in the natural environment. We also determined the optimal screen distance to minimize discomfort due to the vergence-accommodation conflict, and the optimal nasal-temporal positioning of head-mounted display (HMD) screens to maximize binocular field of view. Finally, in a user study we investigated how VR content affects comfort and performance. Content that is more consistent with the statistics of the natural world yields less discomfort than content that is not. Furthermore, consistent content yields slightly better performance than inconsistent content.

Super multi-view near-eye virtual reality with directional backlights from wave-guides

Directional backlights have often been employed for generating multiple view-zones in three-dimensional (3D) display, with each backlight converging into a corresponding view-zone. By designing the view-zone interval for each pupil smaller than the pupil's diameter, super multi-view (SMV) can get implemented for a VAC-free 3D display. However, expanding the backlight from a light-source to cover the corresponding display panel often needs an extra thickness, which results in a thicker structure and is unwanted by a near-eye display. In this paper, two wave-guides are introduced into a near-eye virtual reality (NEVR) system, for sequentially guiding more than one directional backlight to each display panel for SMV display without bringing obvious extra thickness. A prototype SMV NEVR gets demonstrated, with two backlights from each wave-guide converging into two view-zones for a corresponding pupil. Although the additional configured light-sources are positioned far from the corresponding wave-guide in our proof-of-concept prototype, multiple light-sources can be attached to the corresponding wave-guide compactly if necessary. As proof, a 3D scene with defocus-blur effects gets displayed. The design range of the backlights' total reflection angles in the wave-guide is also discussed.

Super multi-view near-eye display with a lightguide combiner

We propose a lightguide-type super multi-view near-eye display that uses a digital micromirror device and a LED array. The proposed method presents three-dimensional images with a natural monocular depth cue using a compact combiner optics which consists of a thin lightguide and holographic optical elements (HOEs). Feasibility of the proposed method is verified by optical experiments which demonstrate monocular three-dimensional image presentation over a wide depth range. We also analyze the degradation of the image quality stemming from the spectral spread of the HOEs and show its reduction by a pre-compensation exploiting an adaptive moment estimation (Adam) optimizer.

Holographic super multi-view Maxwellian near-eye display with eyebox expansion

A holographic super multi-view (SMV) Maxwellian display based on flexible wavefront modulation is proposed for the first time, to the best of our knowledge. It solves the issue that the previous holographic Maxwellian displays could not provide depth cues for monocular vision. Different from the previous methods, two or more parallax images are multiplied by quadric phase distributions and converged to the viewpoints existing in the pupil to provide 3-D vision. A time division method is proposed to eliminate the cross talk caused by the coherence of different spherical waves. Experiments demonstrate that the proposed method can accurately reconstruct images at different depth without cross talk. The proposed method inherits the previous holographic Maxwellian display's advantages of flexible viewpoint position adjustment and large depth of field (DOF). Superior to geometric optics based SMV displays, the proposed system is compact without lens aberration since only a single spatial light modulator (SLM) is needed without any additional optical elements.

Aspherical liquid crystal lenses based on a variable transmission electrode

In this work, a technique to generate aspherical liquid crystal lenses with positive and negative optical power is experimentally demonstrated. The main enabling element is a micro-metric electrode with variable spatial size. This produces a decreasing resistance towards the lens centre that generates the desired voltage/phase profiles. Then, the voltage is homogeneously distributed across the active area of the lens by micro-metric concentric electrodes. As it is demonstrated, the phase shift can be controlled with voltages from 0 to 4.5 V_RMS. As a result, parabolic profiles are obtained both for negative and positive optical powers. Furthermore, this approach avoids some disadvantages of previous techniques; parabolic profiles can be obtained with only one lithographic step and one or two voltage sources. Other complex aspherical profiles could be fabricated using the same technique, such as elliptical or hyperbolic ones.

Simplified retinal 3D projection rendering method and system

A simplified rendering method and system for retinal 3D projection using view and depth information is proposed and demonstrated. Instead of vertex calculations, image-based techniques, including sub-image shifting, image fusion, and hole filling, combined with the depth information, are used to render the multi-view images in a display space with specific discrete depth coordinates. A set of time-division multiplexing retinal 3D projection systems with dense viewpoints is built. A near-eye display of a 3D scene with complex occlusion relationships is realized using the rendering method and system. The eye box of the retinal projection system is enlarged, and the accommodation response of the eyes is evoked at the same time, which improves the visual experience. Rendering tests are carried out using simple and complex models, which proves the effectiveness of this method. Comparative experiments prove that the proposed retinal projection method can obtain high-performance 3D images comparable to the super multi-view display method while simplifying the rendering process. Additionally, the depth of field of the experimental system can cover most of the vergence accommodation conflict sensitive range of the human eye.

Fatigue-free visual perception of high-density super-multiview augmented reality images

It is well known that wearing virtual reality (VR) and augmented reality (AR) devices for long periods can cause visual fatigue and motion sickness due to vergence-accommodation conflict (VAC). VAC is considered the main obstacle to the development of advanced three-dimensional VR and AR technology. In this paper, we present a novel AR high-density super-multiview (HDSMV) display technique capable of eliminating VAC in wide range. The designed binocular time-sequential AR HDSMV projection, which delivers 11 views to each eye pupil, is experimentally demonstrated, confirming that VAC is eliminated over a wide-range of viewer's focus distance. It is believed that the proposed time-sequential AR HDSMV method will pave the way for the development of VAC-free AR technology.

Polarization enlargement of FOV in Super Multi-view display based on near-eye timing-apertures

With strip-type timing-apertures attached to each eye of a viewer, more than one perspective views can be guided to either eye sequentially through different timing-apertures, thus implementing VAC-free (vergence-accommodation conflict-free) SMV (Super Multi-view) 3D (three-dimensional) display. To overcome the FOV (field of view) limitation problem due to small size of the timing-apertures along their arrangement direction, novel polarization architectures are designed to the timing-apertures in this paper. Correspondingly, the display screen of the proposed SMV display system is divided into M > 1 sub-screens along the arrangement direction of the timing-apertures, with adjacent sub-screens emitting light of mutually orthogonal polarization. At a time-point of each time period, a group of M timing-apertures, which correspond to the M sub-screens in a one-by-one manner along the arrangement direction, are turned on for creating an M-fold FOV, with each polarized timing-aperture of the group allowing light from the corresponding sub-screen passing through and blocking light from sub-screen(s) adjacent to the corresponding sub-screen. At 2T > 1 time-points of each time period, 2T groups of timing-apertures are turned on sequentially for presenting more than one two-dimensional images of the displayed scene to each eye, to implement SMV display based on persistence of vision. M stands for the FOV magnification number and T stands for the two-dimensional image number for each eye. As proof, a 3-fold FOV of 41° gets implemented experimentally with a currently available timing-aperture array of M = 3, accompanied by an effective noise-free region (ENFR) of 8.34 mm. Furthermore, the promising of freeing FOV from timing-aperture constraint fundamentally by larger M is described, out-of-screen blur along strip direction of the timing-apertures and the problem of limited ENFR are discussed.

Holographic near-eye display based on complex amplitude modulation with band-limited zone plates

A holographic near-eye display (NED) system based on complex amplitude modulation (CAM) with band-limited zone plates is proposed. The whole system mainly consists of a phase-only spatial light modulator (SLM), an Abbe-Porter filter system, an eyepiece, and an image combiner. The point source method based on band limited zone plates is used to accurately control the bandwidth of the target complex amplitude. The effects of intensity modulation coefficient γ in the frequency-filtering method on the intensity and the quality of reconstructed images are analyzed, which provide a judgment basis for selecting the appropriate value of γ. We also derive the expressions of the field of view (FOV) and exit pupil of the NED system. Since the holographic image is magnified in two steps in this system, the large FOV can be obtained. The optical experimental results show that the proposed system can provide a dynamic holographic three-dimensional (3D) augmented reality (AR) display with a 23.5° horizontal FOV.

Time-division and color multiplexing light-field display using liquid-crystal display panels to induce focal accommodation

A super-multiview light-field display with horizontal and vertical parallax is realized by time-division and color multiplexing to deliver full-color images to each viewpoint. In the conventional study, an image of a different color is delivered to each viewpoint to induce focal accommodation. In the proposed method, we deliver images of different colors sequentially to generate a full-color image by an after-image effect. Though the number of time-divisions increases in the proposed method, perceived flicker is suppressed by showing different colors at different timings. We compare the observed images given by the proposed method with those given by the conventional method to find out that the former reproduces a natural blur effect when the image is defocused. We also confirm with a psychophysical experiment using a refractometer that the proposed method induces a stronger focal accommodation than other super-multiview methods with a smaller number of time-divisions or with a stronger flicker. The proposed optics is applicable to a near-eye display with a natural focal effect.

Eye-box extended retinal projection type near-eye display with multiple independent viewpoints [Invited]

We introduce an approach to expand the eye-box in a retinal-projection-based near-eye display. The retinal projection display has the advantage of providing clear images in a wide depth range; however, it has difficulty in practical use with a narrow eye-box. Here, we propose a method to enhance the eye-box of the retinal projection display by generating multiple independent viewpoints, maintaining a wide depth of field. The method prevents images projected from multiple viewpoints from overlapping one other in the retina. As a result, our proposed system can provide a continuous image over a wide viewing angle without an eye tracker or image update. We discuss the optical design for the proposed method and verify its feasibility through simulation and experiment.

Applications of augmented reality in ophthalmology [Invited]

Throughout the last decade, augmented reality (AR) head-mounted displays (HMDs) have gradually become a substantial part of modern life, with increasing applications ranging from gaming and driver assistance to medical training. Owing to the tremendous progress in miniaturized displays, cameras, and sensors, HMDs are now used for the diagnosis, treatment, and follow-up of several eye diseases. In this review, we discuss the current state-of-the-art as well as potential uses of AR in ophthalmology. This review includes the following topics: (i) underlying optical technologies, displays and trackers, holography, and adaptive optics; (ii) accommodation, 3D vision, and related problems such as presbyopia, amblyopia, strabismus, and refractive errors; (iii) AR technologies in lens and corneal disorders, in particular cataract and keratoconus; (iv) AR technologies in retinal disorders including age-related macular degeneration (AMD), glaucoma, color blindness, and vision simulators developed for other types of low-vision patients.

1000-volume/s high-speed volumetric display for high-speed HMD

In this paper, we propose a high-speed volumetric display principle that can solve two problems faced by three-dimensional displays using the parallax stereo principle (namely, the vergence-accommodation conflict and display latency) and we report evaluation results. The proposed display method can update a set of images at different depths at 1000 Hz and is consistent with accommodation. The method selects the depth position in microseconds by combining a high-speed variable-focus lens that vibrates at about 69 kHz and sub-microsecond control of illumination light using an LED. By turning on the LED for only a few hundred nanoseconds when the refractive power of the lens is at a certain value, an image can be presented with this specific refractive power. The optical system is combined with a DMD to form an image at each depth. 3D information consisting of multiple planes in the depth direction can be presented at a high refresh rate by switching the images and changing the refractive power at high speed. A proof-of-concept system was developed to show the validity of the proposed display principle. The system successfully displayed 3D information consisting of six binary images at an update rate of 1000 volume/s.

Approximated super multi-view head-mounted display to reduce visual fatigue

To reduce the visual fatigue of the head-mounted displays, we propose an approximated super multi-view technique in which multiple viewpoints are generated two-dimensionally with an interval smaller than the pupil diameter using the time multiplexing technique, and left and right virtual images are two-dimensionally shifted in synchronization with the viewpoint generation. The proposed technique enlarges the depth of field of eyes to provide the accommodation-invariant feature so that the vergence-accommodation conflict is mitigated. We constructed an experimental system by using two LED arrays for the viewpoint generation and one LCD panel vibrated by two stepping motors. The proposed technique was then experimentally validated.

Retinal image quality in near-eye pupil-steered systems

State-of-the-art near-eye displays often compromise on eye box size to maintain a wide field of view, necessitating a means for steering the eye box to maintain alignment with a moving eye. The design space of such pupil-steered systems is not well defined and the implications of imperfect steering on the perceived image are not well understood. To better characterize the pupil steering design space, we introduce a generalized taxonomy of pupil-steered architectures that considers both system and ocular factors that affect steering performance. We also develop an optical model of a generalized pupil-steered system with a wide-field schematic eye to simulate the retinal image. Using this framework, we systematically characterize retinal image quality for different combinations of design parameters. The results of these simulations provide an overview of the pupil steering design space and help determine relevant psychophysical experiments for further evaluation.

Demonstration of a low-crosstalk super multi-view light field display with natural depth cues and smooth motion parallax

Due to lack of the accommodation stimulus, an inherent drawback for the conventional glasses-free stereoscopic display is that precise depth cues for the human monocular vision is rent, which results in the well-known convergence-accommodation conflict for the human visual system. Here, a super multi-view light field display with the vertically-collimated programmable directional backlight (VC-PDB) and the light control module (LCM) is demonstrated. The VC-PDB and the LCM are used to form the super multi-view light field display with low crosstalk, which can provide precisely detectable accommodation depth for human monocular vision. Meanwhile, the VC-PDB cooperates with the refreshable liquid-crystal display panel to provide the convergence depth matching the accommodation depth. In addition, the proposed method of light field pick-up and reconstruction is implemented to ensure the perceived three dimensional (3D) images with accurate depth cues and correct geometric occlusion, and the eye tracker is used to enlarge the viewing angle of 3D images with smooth motion parallax. In the experiments, the reconstructed high quality fatigue-free 3D images can be perceived with the clear focus depth of 13 cm in the viewing angle of ± 20°, where 352 viewpoints with the viewpoint density of 1 mm^-1 and the crosstalk of less than 6% are presented.

Towards a Switchable AR/VR Near-eye Display with Accommodation-Vergence and Eyeglass Prescription Support

In this paper, we present our novel design for switchable AR/VR near-eye displays which can help solve the vergence-accommodation-conflict issue. The principal idea is to time-multiplex virtual imagery and real-world imagery and use a tunable lens to adjust focus for the virtual display and the see-through scene separately. With this novel design, prescription eyeglasses for near- and far-sighted users become unnecessary. This is achieved by integrating the wearer's corrective optical prescription into the tunable lens for both virtual display and see-through environment. We built a prototype based on the design, comprised of micro-display, optical systems, a tunable lens, and active shutters. The experimental results confirm that the proposed near-eye display design can switch between AR and VR and can provide correct accommodation for both.

Light field display with ellipsoidal mirror array and single projector

We present a method to create light field display using a single projector and an array of plane mirrors. Mirrors can reproduce densely arranged virtual projectors regardless of the physical size of the real projector, thus producing a light field display of competitive ray density. We propose an ellipsoidal geometric framework and a design pipeline, and use parametric modelling technique to automatically generate the display configurations satisfying target design parameters. Three units of mirror array light field display systems have been implemented to evaluate the proposed methodologies. More importantly, we have experimentally verified that the high-density light field produced by our method can naturally evoke accommodation of the eyes, thereby reducing the vergence-accommodation conflict. The mirror array approach allows flexible trading between the spatial and angular resolutions for accommodating different applications, thus providing a practical solution to realize projection-based light field display.