Stereoscopic 3D Visual Discomfort Prediction: A Dynamic Accommodation and Vergence Interaction Model

Autostereoscopic 3D viewing can change the dimensions of the crystalline lens in myopes

PURPOSE

Autostereoscopic displays have become increasingly common, but their impact on ocular dimensions remains unknown. We sought to identify changes in the crystalline lens dimensions induced by autostereoscopic three-dimensional (3D) viewing.

METHODS

Forty young adults (age: 22.6 ± 2.0 years, male/female: 15/25) were consecutively enrolled and randomly divided into two groups (3D and two-dimensional [2D] viewing groups) to watch a 30-min movie clip displayed in 3D or 2D mode on a tablet computer. The lens thickness (LT), diameter, curvature, decentration and tilt were measured with anterior segment optical coherence tomography under both non-accommodating (static) and accommodating conditions.

RESULTS

In the static condition, the LT decreased by 0.03 ± 0.03 mm (p < 0.001) and the anterior radius of curvature (ARC) increased by 0.49 ± 0.59 mm (p = 0.001) post-3D viewing. In contrast, following 2D viewing, the ARC decreased by 0.23 ± 0.25 mm (p = 0.001). Additionally, the increase in the steep ARC post-3D viewing was greater in high-myopic eyes than low to moderate myopic eyes (p = 0.04). When comparing the accommodative with the static (non-accommodative) condition, for 3D viewing the lens decentration decreased (-0.03 ± 0.05 mm, p = 0.02); while for 2D viewing, the posterior curvature radius (-0.14 ± 0.20 mm, p = 0.006) and diameter (-0.13 ± 0.20 mm, p = 0.01) decreased.

CONCLUSIONS

Viewing with the autostereoscopic 3D tablet could temporally decrease the thickness and curvature of the lens under non-accommodating conditions. However, its long-term effect requires further exploration.

Cybersickness and Its Severity Arising from Virtual Reality Content: A Comprehensive Study

Virtual reality (VR) experiences often elicit a negative effect, cybersickness, which results in nausea, disorientation, and visual discomfort. To quantitatively analyze the degree of cybersickness depending on various attributes of VR content (i.e., camera movement, field of view, path length, frame reference, and controllability), we generated cybersickness reference (CYRE) content with 52 VR scenes that represent different content attributes. A protocol for cybersickness evaluation was designed to collect subjective opinions from 154 participants as reliably as possible in conjunction with objective data such as rendered VR scenes and biological signals. By investigating the data obtained through the experiment, the statistically significant relationships—the degree that the cybersickness varies with each isolated content factor—are separately identified. We showed that the cybersickness severity was highly correlated with six biological features reflecting brain activities (i.e., relative power spectral densities of Fp1 delta, Fp 1 beta, Fp2 delta, Fp2 gamma, T4 delta, and T4 beta waves) with a coefficient of determination greater than 0.9. Moreover, our experimental results show that individual characteristics (age and susceptibility) are also quantitatively associated with cybersickness level. Notably, the constructed dataset contains a number of labels (i.e., subjective cybersickness scores) that correspond to each VR scene. We used these labels to build cybersickness prediction models and obtain a reliable predictive performance. Hence, the proposed dataset is supposed to be widely applicable in general-purpose scenarios regarding cybersickness quantification.

A Deep Motion Sickness Predictor Induced by Visual Stimuli in Virtual Reality

In a virtual reality (VR) environment, where visual stimuli predominate over other stimuli, the user experiences cybersickness because the balance of the body collapses due to self-motion. Accordingly, the VR experience is accompanied by unavoidable sickness referred to as visually induced motion sickness (VIMS). In this article, our primary purpose is to simultaneously estimate the VIMS score by referring to the content and calculate the temporally induced VIMS sensitivity. To seek our goals, we propose a novel architecture composed of two consecutive networks: 1) neurological representation and 2) spatiotemporal representation. In the first stage, the network imitates and learns the neurological mechanism of motion sickness. In the second stage, the significant feature of the spatial and temporal domains is expressed over the generated frames. After the training procedure, our model can calculate VIMS sensitivity for each frame of the VR content by using the weakly supervised approach for unannotated temporal VIMS scores. Furthermore, we release a massive VR content database. In the experiments, the proposed framework demonstrates excellent performance for VIMS score prediction compared with existing methods, including feature engineering and deep learning-based approaches. Furthermore, we propose a way to visualize the cognitive response to visual stimuli and demonstrate that the induced sickness tends to be activated in a similar tendency, as done in clinical studies.

Deep Multimodal Fusion Autoencoder for Saliency Prediction of RGB-D Images

In recent years, the prediction of salient regions in RGB-D images has become a focus of research. Compared to its RGB counterpart, the saliency prediction of RGB-D images is more challenging. In this study, we propose a novel deep multimodal fusion autoencoder for the saliency prediction of RGB-D images. The core trainable autoencoder of the RGB-D saliency prediction model employs two raw modalities (RGB and depth/disparity information) as inputs and their corresponding eye-fixation attributes as labels. The autoencoder comprises four main networks: color channel network, disparity channel network, feature concatenated network, and feature learning network. The autoencoder can mine the complex relationship and make the utmost of the complementary characteristics between both color and disparity cues. Finally, the saliency map is predicted via a feature combination subnetwork, which combines the deep features extracted from a prior learning and convolutional feature learning subnetworks. We compare the proposed autoencoder with other saliency prediction models on two publicly available benchmark datasets. The results demonstrate that the proposed autoencoder outperforms these models by a significant margin.

Deep Visual Saliency on Stereoscopic Images

Visual saliency on stereoscopic 3D (S3D) images has been shown to be heavily influenced by image quality. Hence, this dependency is an important factor in image quality prediction, image restoration and discomfort reduction, but it is still very difficult to predict such a nonlinear relation in images. In addition, most algorithms specialized in detecting visual saliency on pristine images may unsurprisingly fail when facing distorted images. In this paper, we investigate a deep learning scheme named Deep Visual Saliency (DeepVS) to achieve a more accurate and reliable saliency predictor even in the presence of distortions. Since visual saliency is influenced by low-level features (contrast, luminance and depth information) from a psychophysical point of view, we propose seven low-level features derived from S3D image pairs and utilize them in the context of deep learning to detect visual attention adaptively to human perception. During analysis, it turns out that the low-level features play a role to extract distortion and saliency information. To construct saliency predictors, we weight and model the human visual saliency through two different network architectures, a regression and a fully convolutional neural networks (CNNs). Our results from thorough experiments confirm that the predicted saliency maps are up to 70 % correlated with human gaze patterns, which emphasize the need for the hand-crafted features as input to deep neural networks in S3D saliency detection.

Deep Visual Discomfort Predictor for Stereoscopic 3D Images

Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Towards advancing performance on this problem, we have developed a deep learning based VDP model named Deep Visual Discomfort Predictor (DeepVDP). DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing S3D image into local regions defined as patches, and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patchbased CNN model that has been pretrained on proxy groundtruth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve state-of-the-art performance as compared to previous VDP algorithms.

Computational Imaging and Displays: Capturing and displaying richer representations of the world

Computational imaging techniques allow capturing richer, more complete representations of a scene through the introduction of novel computational algorithms, overcoming the limitations imposed by hardware and optics. The areas of application range from medical imaging to security, to areas in engineering, to name a few. Computational displays combine optics, hardware and computation to faithfully reproduce the world as seen by our eyes, something that current displays still cannot do. This article looks at a few examples of both, aiming to convey the power of the joint co-design of hardware and computational algorithms, also taking into account visual perception.

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.

Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images

Conventional stereoscopic 3D (S3D) displays do not provide accommodation depth cues of the 3D image or video contents being viewed. The sense of content depths is thus limited to cues supplied by motion parallax (for 3D video), stereoscopic vergence cues created by presenting left and right views to the respective eyes, and other contextual and perspective depth cues. The absence of accommodation cues can induce two kinds of accommodation vergence mismatches (AVM) at the fixation and peripheral points, which can result in severe visual discomfort. With the aim of alleviating discomfort arising from AVM, we propose a new visual comfort enhancement approach for processing S3D visual signals to deliver a more comfortable 3D viewing experience at the display. This is accomplished via an optimization process whereby a predictive indicator of visual discomfort is minimized, while still aiming to maintain the viewer's sense of 3D presence by performing a suitable parallax shift, and by directed blurring of the signal. Our processing framework is defined on 3D visual coordinates that reflect the nonuniform resolution of retinal sensors and that uses a measure of 3D saliency strength. An appropriate level of blur that corresponds to the degree of parallax shift is found, making it possible to produce synthetic accommodation cues implemented using a perceptively relevant filter. By this method, AVM, the primary contributor to the discomfort felt when viewing S3D images, is reduced. We show via a series of subjective experiments that the proposed approach improves visual comfort while preserving the sense of 3D presence.

Asymmetrically Compressed Stereoscopic 3D Videos: Quality Assessment and Rate-Distortion Performance Evaluation

Objective quality assessment of stereoscopic 3D video is challenging but highly desirable, especially in the application of stereoscopic video compression and transmission, where useful quality models are missing, that can guide the critical decision making steps in the selection of mixed-resolution coding, asymmetric quantization, and pre- and post-processing schemes. Here we first carry out subjective quality assessment experiments on two databases that contain various asymmetrically compressed stereoscopic 3D videos obtained from mixed-resolution coding, asymmetric transform-domain quantization coding, their combinations, and the multiple choices of postprocessing techniques. We compare these asymmetric stereoscopic video coding schemes with symmetric coding methods and verify their potential coding gains. We observe a strong systematic bias when using direct averaging of 2D video quality of both views to predict 3D video quality. We then apply a binocular rivalry inspired model to account for the prediction bias, leading to a significantly improved full reference quality prediction model of stereoscopic videos. The model allows us to quantitatively predict the coding gain of different variations of asymmetric video compression, and provides new insight on the development of high efficiency 3D video coding schemes.

Visual Presence: Viewing Geometry Visual Information of UHD S3D Entertainment

To maximize the presence experienced by humans, visual content has evolved to achieve a higher visual presence in a series of high definition (HD), ultra HD (UHD), 8K UHD, and 8K stereoscopic 3D (S3D). Several studies have introduced visual presence delivered from content when viewing UHD S3D from a content analysis perspective. Nevertheless, no clear definition has been presented for visual presence, and only a subjective evaluation has been relied upon. The main reason for this is that there is a limitation to defining visual presence via the use of content information itself. In this paper, we define the visual presence for each viewing environment, and investigate a novel methodology to measure the experienced visual presence when viewing both 2D and 3D via the definition of a new metric termed volume of visual information by quantifying the influence of the viewing geometry between the display and viewer. To achieve this goal, the viewing geometry and display parameters for both flat and atypical displays are analyzed in terms of human perception by introducing a novel concept of pixel-wise geometry. In addition, perceptual weighting through analysis of content information is performed in accordance with monocular and binocular vision characteristics. In the experimental results, it is shown that the constructed model based on the viewing geometry, content, and perceptual characteristics has a high correlation of about 84% with subjective evaluations.