See-through integral imaging display with background occlusion capability

Occlusion Handling in Augmented Reality: Past, Present and Future

One of the main goals of many augmented reality applications is to provide a seamless integration of a real scene with additional virtual data. To fully achieve that goal, such applications must typically provide high-quality real-world tracking, support real-time performance and handle the mutual occlusion problem, estimating the position of the virtual data into the real scene and rendering the virtual content accordingly. In this survey, we focus on the occlusion handling problem in augmented reality applications and provide a detailed review of 161 articles published in this field between January 1992 and August 2020. To do so, we present a historical overview of the most common strategies employed to determine the depth order between real and virtual objects, to visualize hidden objects in a real scene, and to build occlusion-capable visual displays. Moreover, we look at the state-of-the-art techniques, highlight the recent research trends, discuss the current open problems of occlusion handling in augmented reality, and suggest future directions for research.

Design of a Pupil-Matched Occlusion-Capable Optical See-Through Wearable Display

State-of-the-art optical see-through head-mounted displays (OST-HMD) for augmented reality applications lack the ability to correctly render light blocking behavior between digital and physical objects, known as mutual occlusion capability. In this article, we present a novel optical architecture for enabling a high performance, occlusion-capable optical see-through head-mounted display (OCOST-HMD). The design utilizes a single-layer, double-pass architecture, creating a compact OCOST-HMD that is capable of rendering per-pixel mutual occlusion, correctly pupil-matched viewing perspective between virtual and real scenes, and a wide see-through field of view (FOV). Based on this architecture, we present a design embodiment and a compact prototype implementation. The prototype demonstrates a virtual display with an FOV of 34° by 22°, an angular resolution of 1.06 arc minutes per pixel, and an average image contrast greater than 40 percent at the Nyquist frequency of 53 cycles/mm. Furthermore, the device achieves a see-through FOV of 90° by 50°, within which about 40° diagonally is occlusion-enabled, and has an angular resolution of 1.0 arc minutes (comparable to a 20/20 vision) and a dynamic range greater than 100:1. We conclude the paper with a quantitative comparison of the key optical performance such as modulation transfer function, image contrast, and color rendering accuracy of our OCOST-HMD system with and without occlusion enabled for various lighting environments.

Flicker-free dual-volume augmented reality display using a pixelated interwoven integral floating technique with a geometric phase lens

A geometric phase (GP) integral floating display can provide multifocal three-dimensional (3D) augmented reality (AR) images with enhanced depth expression by switching the focal modes of the GP lens via polarization control. However, using temporal multiplexing to switch between the focal modes of GP optics causes flickering as each 3D AR image is fully presented in different frames and their temporal luminance profile becomes easily recognizable, particularly as the number of available focal modes increases. Here, we propose a novel integral floating technique to generate pixelated interwoven 3D AR images; a half of each image is spatially mixed with another and presented in both focal modes simultaneously to resolve the flickering issue. The principle was verified via experimental demonstration and optically measured data.

Focal surface occlusion

This paper proposes focal surface occlusion to provide focal cues of occlusion masks for multiple virtual objects at continuous depths in an occlusion-capable optical see-through head-mounted display. A phase-only spatial light modulator (PSLM) that acts as a dynamic free-form lens is used to conform the focal surface of an occlusion mask to the geometry of the virtual scene. To reproduce multiple and continuous focal blurs while reducing the distortion of the see-through view, an optical design based on afocal optics and edge-based optimization to exploit a property of the occlusion mask is established. The prototype with the PSLM and transmissive liquid crystal display can reproduce the focus blur of occluded objects at multiple and continuous depths with a field of view of 14.6°.

Occlusion-capable see-through display without the screen-door effect using a photochromic mask

Conventional occlusion-capable see-through display systems have many practical limitations such as the form factor, narrow field of view, screen-door effect, and diffraction of a real scene. In this Letter, we propose an occlusion-capable see-through display using lens arrays and a photochromic plate. By imaging the occlusion mask on the photochromic plate with near-UV light, the visible light transmittance of the plate changes. Since no black matrix lies on the photochromic plate, our system provides a clear real scene view without the grid structure of the pixels and can prevent diffraction defects of the real scene. We also alleviate the drawback of a narrow field of view using the lens arrays for a reduced form factor.

LensIet VR: Thin, Flat and Wide-FOV Virtual Reality Display Using Fresnel Lens and LensIet Array

We propose a new thin and flat virtual reality (VR) display design using a Fresnel lenslet array, a Fresnel lens, and a polarization-based optical folding technique. The proposed optical system has a wide field of view (FOV) of 102°x102°, a wide eye-box of 8.8 mm, and an ergonomic eye-relief of 20 mm. Simultaneously, only 3.3 mm of physical distance is required between the display panel and the lens, so that the integrated VR display can have a compact form factor like sunglasses. Moreover, since all lenslet of the lenslet array is designed to operate under on-axis condition with low aberration, the discontinuous pupil swim distortion between the lenslets is hardly observed. In addition, all on-axis lenslets can be designed identically, reducing production cost, and even off-the-shelf Fresnel optics can be used. In this paper, we introduce how we design system parameters and analyze system performance. Finally, we demonstrate two prototypes and experimentally verify that the proposed VR display system has the expected performance while having a glasses-like form factor.

StainedView: Variable-Intensity Light-Attenuation Display with Cascaded Spatial Color Filtering for Improved Color Fidelity

We present StainedView, an optical see-through display that spatially filters the spectral distribution of light to form an image with improved color fidelity. Existing light-attenuation displays have limited color fidelity and contrast, resulting in a degraded appearance of virtual images. To use these displays to present virtual images that are more consistent with the real world, we require three things: intensity modulation of incoming light, spatial color filtering with narrower bandwidth, and appropriate light modulation for incoming light with an arbitrary spectral distribution. In StainedView, we address the three requirements by cascading two phase-only spatial light modulators (PSLMs), a digital micromirror device, and polarization optics to control both light intensity and spectrum distribution. We show that our design has a 1.8 times wider color gamut fidelity (75.8% fulfillment of sRGB color space) compared to the existing single-PSLM approach (41.4%) under a reference white light. We demonstrated the design with a proof-of-concept display system. We further introduce our optics design and pixel-selection algorithm for the given light input, evaluate the spatial color filter, and discuss the limitation of the current prototype.

Compensation of color breaking in bi-focal depth-switchable integral floating augmented reality display with a geometrical phase lens

A bi-focal integral floating system using a geometrical phase (GP) lens can provide switchable integrated spaces with enhanced three-dimensional (3D) augmented reality (AR) depth expression. However, due to the chromatic aberration properties of the GP lens implemented for the switchable depth-floating 3D images, the floated 3D AR images with the red/green/blue (R/G/B) colors are formed at different depth locations with different magnification effects, which causes color breaking. In this paper, we propose a novel technique to resolve the color breaking problem by integrating the R/G/B elemental images with compensated depths and sizes along with experiments to demonstrate the improved results. When we evaluated the color differences of the floated 3D AR images based on CIEDE2000, the experimental results of the depth-switchable integral floating 3D AR images showed that the color accuracies were greatly improved after applying a pre-compensation scheme to the R/G/B sub-images in both concave and convex lens operation modes of the bi-focal switching GP floating lens.

Occlusion-capable optical-see-through near-eye display using a single digital micromirror device

Occlusion of a real scene by displayed virtual images mitigates incorrect depth cues and enhances image visibility in augmented reality applications. In this Letter, we propose a novel optical scheme for the occlusion-capable optical-see-through near-eye display. The proposed scheme uses only a single spatial light modulator, as the real-scene mask and virtual image display simultaneously. A polarization-based double-pass configuration is also combined, enabling a compact implementation. The proposed scheme is verified by optical experiments which demonstrate a 60 Hz red-green-blue video display with a 4-bit depth for each color channel and per-pixel dynamic occlusion of a 90.6% maximum occlusion ratio.

Image quality improvement of holographic 3-D images based on a wavefront recording plane method with a limiting diffraction region

This study aims to improve the image quality of holographic three-dimensional (3-D) images based on the wavefront recording plane (WRP) method. In this method, we place a WRP close to the 3-D objects to reduce the propagation distance of light from the objects to the WRP. The conventional WRP method has been implemented only under conditions that did not cause aliasing noise. This study proposes a WRP method with a limiting diffraction region from the WRP to the hologram such that we can perform the WRP method under any condition. As a result, we succeeded in improving the image quality of the 3-D images based on the WRP method.

Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display

Occlusion is a powerful visual cue that is crucial for depth perception and realism in optical see-through augmented reality (OST-AR). However, existing OST-AR systems additively overlay physical and digital content with beam combiners - an approach that does not easily support mutual occlusion, resulting in virtual objects that appear semi-transparent and unrealistic. In this work, we propose a new type of occlusion-capable OST-AR system. Rather than additively combining the real and virtual worlds, we employ a single digital micromirror device (DMD) to merge the respective light paths in a multiplicative manner. This unique approach allows us to simultaneously block light incident from the physical scene on a pixel-by-pixel basis while also modulating the light emitted by a light-emitting diode (LED) to display digital content. Our technique builds on mixed binary/continuous factorization algorithms to optimize time-multiplexed binary DMD patterns and their corresponding LED colors to approximate a target augmented reality (AR) scene. In simulations and with a prototype benchtop display, we demonstrate hard-edge occlusions, plausible shadows, and also gaze-contingent optimization of this novel display mode, which only requires a single spatial light modulator.

Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics

Optical see-through augmented reality (AR) systems are a next-generation computing platform that offer unprecedented user experiences by seamlessly combining physical and digital content. Many of the traditional challenges of these displays have been significantly improved over the last few years, but AR experiences offered by today's systems are far from seamless and perceptually realistic. Mutually consistent occlusions between physical and digital objects are typically not supported. When mutual occlusion is supported, it is only supported for a fixed depth. We propose a new optical see-through AR display system that renders mutual occlusion in a depth-dependent, perceptually realistic manner. To this end, we introduce varifocal occlusion displays based on focus-tunable optics, which comprise a varifocal lens system and spatial light modulators that enable depth-corrected hard-edge occlusions for AR experiences. We derive formal optimization methods and closed-form solutions for driving this tunable lens system and demonstrate a monocular varifocal occlusion-capable optical see-through AR display capable of perceptually realistic occlusion across a large depth range.

Varifocal Occlusion for Optical See-Through Head-Mounted Displays using a Slide Occlusion Mask

We propose a varifocal occlusion technique for optical see-through head-mounted displays (OST-HMDs). Occlusion in OST-HMDs is a powerful visual cue that enables depth perception in augmented reality (AR). Without occlusion, virtual objects rendered by an OST-HMD appear semi-transparent and less realistic. A common occlusion technique is to use spatial light modulators (SLMs) to block incoming light rays at each pixel on the SLM selectively. However, most of the existing methods create an occlusion mask only at a single, fixed depth-typically at infinity. With recent advances in varifocal OST-HMDs, such traditional fixed-focus occlusion causes a mismatch in depth between the occlusion mask plane and the virtual object to be occluded, leading to an uncomfortable user experience with blurred occlusion masks. In this paper, we thus propose an OST-HMD system with varifocal occlusion capability: we physically slide a transmissive liquid crystal display (LCD) to optically shift the occlusion plane along the optical path so that the mask appears sharp and aligns to a virtual image at a given depth. Our solution has several benefits over existing varifocal occlusion methods: it is computationally less demanding and, more importantly, it is optically consistent, i.e., when a user loses focus on the corresponding virtual image, the mask again gets blurred consistently as the virtual image does. In the experiment, we build a proof-of-concept varifocal occlusion system implemented with a custom retinal projection display and demonstrate that the system can shift the occlusion plane to depths ranging from 25 cm to infinity.

Light Attenuation Display: Subtractive See-Through Near-Eye Display via Spatial Color Filtering

We present a display for optical see-through near-eye displays based on light attenuation, a new paradigm that forms images by spatially subtracting colors of light. Existing optical see-through head-mounted displays (OST-HMDs) form virtual images in an additive manner-they optically combine the light from an embedded light source such as a microdisplay into the users' field of view (FoV). Instead, our light attenuation display filters the color of the real background light pixel-wise in the users' see-through view, resulting in an image as a spatial color filter. Our image formation is complementary to existing light-additive OST-HMDs. The core optical component in our system is a phase-only spatial light modulator (PSLM), a liquid crystal module that can control the phase of the light in each pixel. By combining PSLMs with polarization optics, our system realizes a spatially programmable color filter. In this paper, we introduce our optics design, evaluate the spatial color filter, consider applications including image rendering and FoV color control, and discuss the limitations of the current prototype.

Twin imaging phenomenon of integral imaging

The imaging principles and phenomena of integral imaging technique have been studied in detail using geometrical optics, wave optics, or light filed theory. However, most of the conclusions are only suit for the integral imaging systems using diffused illumination. In this work, a kind of twin imaging phenomenon and mechanism has been observed in a non-diffused illumination reflective integral imaging system. Interactive twin images including a real and a virtual 3D image of one object can be activated in the system. The imaging phenomenon is similar to the conjugate imaging effect of hologram, but it base on the refraction and reflection instead of diffraction. The imaging characteristics and mechanisms different from traditional integral imaging are deduced analytically. Thin film integral imaging systems with 80μm thickness have also been made to verify the imaging phenomenon. Vivid lighting interactive twin 3D images have been realized using a light-emitting diode (LED) light source. When the LED is moving, the twin 3D images are moving synchronously. This interesting phenomenon shows a good application prospect in interactive 3D display, argument reality, and security authentication.

Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive Spatial Light Modulator

We propose an occlusion compensation method for optical see-through head-mounted displays (OST-HMDs) equipped with a singlelayer transmissive spatial light modulator (SLM), in particular, a liquid crystal display (LCD). Occlusion is an important depth cue for 3D perception, yet realizing it on OST-HMDs is particularly difficult due to the displays' semitransparent nature. A key component for the occlusion support is the SLM-a device that can selectively interfere with light rays passing through it. For example, an LCD is a transmissive SLM that can block or pass incoming light rays by turning pixels black or transparent. A straightforward solution places an LCD in front of an OST-HMD and drives the LCD to block light rays that could pass through rendered virtual objects at the viewpoint. This simple approach is, however, defective due to the depth mismatch between the LCD panel and the virtual objects, leading to blurred occlusion. This led existing OST-HMDs to employ dedicated hardware such as focus optics and multi-stacked SLMs. Contrary to these viable, yet complex and/or computationally expensive solutions, we return to the single-layer LCD approach for the hardware simplicity while maintaining fine occlusion-we compensate for a degraded occlusion area by overlaying a compensation image. We compute the image based on the HMD parameters and the background scene captured by a scene camera. The evaluation demonstrates that the proposed method reduced the occlusion leak error by 61.4% and the occlusion error by 85.7%.

Asymmetric integral imaging system for a see-through three-dimensional display with background imaging function

A see-through three-dimensional display with variable background imaging function is proposed. The proposed display system is based on integral imaging and consists of three lens arrays and a transparent flat-panel display. An asymmetric alignment of the three lens arrays enables variable background imaging. The background scene situated at any distance from the display system can be imaged at an intended distance from the display system. The possible imaging regions are shown. The proposed technique was experimentally verified using two optical systems that consisted of lens arrays with large and small lens pitches.