Visual Presence: Viewing Geometry Visual Information of UHD S3D Entertainment

From Human Pose Similarity Metric to 3D Human Pose Estimator: Temporal Propagating LSTM Networks

Predicting a 3D pose directly from a monocular image is a challenging problem. Most pose estimation methods proposed in recent years have shown 'quantitatively' good results (below  ∼ 50mm). However, these methods remain 'perceptually' flawed because their performance is only measured via a simple distance metric. Although this fact is well understood, the reliance on 'quantitative' information implies that the development of 3D pose estimation methods has been slowed down. To address this issue, we first propose a perceptual Pose SIMilarity (PSIM) metric, by assuming that human perception (HP) is highly adapted to extracting structural information from a given signal. Second, we present a perceptually robust 3D pose estimation framework: Temporal Propagating Long Short-Term Memory networks (TP-LSTMs). Toward this, we analyze the information-theory-based spatio-temporal posture correlations, including joint interdependency, temporal consistency, and HP. The experimental results clearly show that the proposed PSIM metric achieves a superior correlation with users' subjective opinions than conventional pose metrics. Furthermore, we demonstrate the significant quantitative and perceptual performance improvements of TP-LSTMs compared to existing state-of-the-art methods.

Tackling Explicit Material from Online Video Conferencing Software for Education Using Deep Attention Neural Architectures

The spread of the COVID-19 pandemic affected all areas of social life, especially education. Globally, many states have closed schools temporarily or imposed local curfews. According to UNESCO estimations, approximately 1.5 billion students have been affected by the closure of schools and the mandatory implementation of distance learning. Although rigorous policies are in place to ban harmful and dangerous content aimed at children, there are many cases where minors, mainly students, have been exposed relatively or unfairly to inappropriate, especially sexual content, during distance learning. Ensuring minors' emotional and mental health is a priority for any education system. This paper presents a severe attention neural architecture to tackle explicit material from online education video conference applications to deal with similar incidents. This is an advanced technique that, for the first time in the literature, proposes an intelligent mechanism that, although it uses attention mechanisms, does not have a square complexity of memory and time in terms of the size of the input. Specifically, we propose the implementation of a Generative Adversarial Network (GAN) with the help of a local, sparse attention mechanism, which can accurately detect obscene and mainly sexual content in streaming online video conferencing software for education.

VR Sickness Versus VR Presence: A Statistical Prediction Model

Although it is well-known that the negative effects of VR sickness, and the desirable sense of presence are important determinants of a user's immersive VR experience, there remains a lack of definitive research outcomes to enable the creation of methods to predict and/or optimize the trade-offs between them. Most VR sickness assessment (VRSA) and VR presence assessment (VRPA) studies reported to date have utilized simple image patterns as probes, hence their results are difficult to apply to the highly diverse contents encountered in general, real-world VR environments. To help fill this void, we have constructed a large, dedicated VR sickness/presence (VR-SP) database, which contains 100 VR videos with associated human subjective ratings. Using this new resource, we developed a statistical model of spatio-temporal and rotational frame difference maps to predict VR sickness. We also designed an exceptional motion feature, which is expressed as the correlation between an instantaneous change feature and averaged temporal features. By adding additional features (visual activity, content features) to capture the sense of presence, we use the new data resource to explore the relationship between VRSA and VRPA. We also show the aggregate VR-SP model is able to predict VR sickness with an accuracy of 90% and VR presence with an accuracy of 75% using the new VR-SP dataset.

Curved TVs improved watching experience when display curvature radii approached viewing distances: Effects of display curvature radius, viewing distance, and lateral viewing position on TV watching experience

Although watching TV often involves multiple viewing distances and viewers, less attention has been paid to the effects of display curvature radius, viewing distance, and lateral viewing position on TV watching experience. This study examined the effects of four display curvature radii (2300R, 4000R, 6000R, and flat), two viewing distances (2.3 m and 4 m), and five lateral viewing positions (P₁-P₅; 0, 35, 70, 105, and 140 cm off-center) on seven TV watching experience elements (spatial presence, engagement, ecological validity, negative effects, visual comfort, image quality, and user satisfaction). Fifty-six individuals (14 per display curvature radius) were seated in pairs to watch videos, each time at a different viewing position (2 viewing distances × 5 paired lateral viewing positions). The spatial presence and engagement increased when display curvature radius approached a viewing distance and lateral viewing position approached P₁, with 4000R-4m-P₁ (display curvature radius-viewing distance-lateral viewing position) providing the best results. Lateral viewing position alone significantly affected five TV watching experience elements; the spatial presence and engagement decreased at P₃-P₅, and ecological validity, image quality, and user satisfaction decreased at P₄-P₅. However, display curvature radius alone did not appreciably affect TV watching experience, and viewing distance alone significantly affected visual comfort only, with a 4-m viewing distance increasing visual comfort. This study demonstrated that effective display curvature radii for watching TV are viewing distance-dependent, and less off-center lateral viewing positions (P₁-P₂) are recommended for TV watching experience. Finally, among the TV watching experience elements, engagement explained user satisfaction to the greatest degree.

Unified No-Reference Quality Assessment of Singly and Multiply Distorted Stereoscopic Images

A challenging problem in the no-reference quality assessment of multiply distorted stereoscopic images (MDSIs) is to simulate the monocular and binocular visual properties under a mixed type of distortions. Due to the joint effects of multiple distortions in MDSIs, the underlying monocular and binocular visual mechanisms have different manifestations with those of singly distorted stereoscopic images (SDSIs). This paper presents a unified no-reference quality evaluator for SDSIs and MDSIs by learning monocular and binocular local visual primitives (MB-LVPs). The main idea is to learn MB-LVPs to characterize the local receptive field properties of the visual cortex in response to SDSIs and MDSIs. Furthermore, we also consider that the learning of primitives should be performed in a task-driven manner. For this, two penalty terms including reconstruction error and quality inconsistency are jointly minimized within a supervised dictionary learning framework, generating a set of quality-oriented MB-LVPs for each single and multiple distortion modality. Given an input stereoscopic image, feature encoding is performed using the learned MB-LVPs as codebooks, resulting in the corresponding monocular and binocular responses. Finally, responses across all the modalities are fused with probabilistic weights which are determined by the modality-specific sparse reconstruction errors, yielding the final monocular and binocular features for quality regression. The superiority of our method has been verified on several SDSI and MDSI databases.

Deep Visual Discomfort Predictor for Stereoscopic 3D Images

Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Towards advancing performance on this problem, we have developed a deep learning based VDP model named Deep Visual Discomfort Predictor (DeepVDP). DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing S3D image into local regions defined as patches, and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patchbased CNN model that has been pretrained on proxy groundtruth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve state-of-the-art performance as compared to previous VDP algorithms.

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.

Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images

Conventional stereoscopic 3D (S3D) displays do not provide accommodation depth cues of the 3D image or video contents being viewed. The sense of content depths is thus limited to cues supplied by motion parallax (for 3D video), stereoscopic vergence cues created by presenting left and right views to the respective eyes, and other contextual and perspective depth cues. The absence of accommodation cues can induce two kinds of accommodation vergence mismatches (AVM) at the fixation and peripheral points, which can result in severe visual discomfort. With the aim of alleviating discomfort arising from AVM, we propose a new visual comfort enhancement approach for processing S3D visual signals to deliver a more comfortable 3D viewing experience at the display. This is accomplished via an optimization process whereby a predictive indicator of visual discomfort is minimized, while still aiming to maintain the viewer's sense of 3D presence by performing a suitable parallax shift, and by directed blurring of the signal. Our processing framework is defined on 3D visual coordinates that reflect the nonuniform resolution of retinal sensors and that uses a measure of 3D saliency strength. An appropriate level of blur that corresponds to the degree of parallax shift is found, making it possible to produce synthetic accommodation cues implemented using a perceptively relevant filter. By this method, AVM, the primary contributor to the discomfort felt when viewing S3D images, is reduced. We show via a series of subjective experiments that the proposed approach improves visual comfort while preserving the sense of 3D presence.