Full-reference quality assessment of stereoscopic images by learning binocular receptive field properties

Dual-Stream Interactive Networks for No-Reference Stereoscopic Image Quality Assessment

The goal of objective stereoscopic image quality assessment (SIQA) is to predict the human perceptual quality of stereoscopic/3D images automatically and accurately. Compared with traditional 2D image quality assessment, the quality assessment of stereoscopic images is more challenging because of complex binocular vision mechanisms and multiple quality dimensions. In this paper, inspired by the hierarchical dual-stream interactive nature of the human visual system, we propose a stereoscopic image quality assessment network (StereoQA-Net) for no-reference stereoscopic image quality assessment. The proposed StereoQA-Net is an end-to-end dual-stream interactive network containing left and right view sub-networks, where the interaction of the two sub-networks exists in multiple layers. We evaluate our method on the LIVE stereoscopic image quality databases. The experimental results show that our proposed StereoQA-Net outperforms state-of-the-art algorithms on both symmetrically and asymmetrically distorted stereoscopic image pairs of various distortion types. In a more general case, the proposed StereoQA-Net can effectively predict the perceptual quality of local regions. In addition, cross-dataset experiments also demonstrate the generalization ability of our algorithm.

Sparse Representation based Video Quality Assessment for Synthesized 3D Videos

The temporal flicker distortion is one of the most annoying noises in synthesized virtual view videos when they are rendered by compressed multi-view video plus depth in Three Dimensional (3D) video system. To assess the synthesized view video quality and further optimize the compression techniques in 3D video system, objective video quality assessment which can accurately measure the flicker distortion is highly needed. In this paper, we propose a full reference sparse representation based video quality assessment method towards synthesized 3D videos. Firstly, a synthesized video, treated as a 3D volume data with spatial (X-Y) and temporal (T) domains, is reformed and decomposed as a number of spatially neighboring temporal layers, i.e., X-T or Y-T planes. Gradient features in temporal layers of the synthesized video and strong edges of depth maps are used as key features in detecting the location of flicker distortions. Secondly, dictionary learning and sparse representation for the temporal layers are then derived and applied to effectively represent the temporal flicker distortion. Thirdly, a rank pooling method is used to pool all the temporal layer scores and obtain the score for the flicker distortion. Finally, the temporal flicker distortion measurement is combined with the conventional spatial distortion measurement to assess the quality of synthesized 3D videos. Experimental results on synthesized video quality database demonstrate our proposed method is significantly superior to other state-of-the-art methods, especially on the view synthesis distortions induced from depth videos.

Unified No-Reference Quality Assessment of Singly and Multiply Distorted Stereoscopic Images

A challenging problem in the no-reference quality assessment of multiply distorted stereoscopic images (MDSIs) is to simulate the monocular and binocular visual properties under a mixed type of distortions. Due to the joint effects of multiple distortions in MDSIs, the underlying monocular and binocular visual mechanisms have different manifestations with those of singly distorted stereoscopic images (SDSIs). This paper presents a unified no-reference quality evaluator for SDSIs and MDSIs by learning monocular and binocular local visual primitives (MB-LVPs). The main idea is to learn MB-LVPs to characterize the local receptive field properties of the visual cortex in response to SDSIs and MDSIs. Furthermore, we also consider that the learning of primitives should be performed in a task-driven manner. For this, two penalty terms including reconstruction error and quality inconsistency are jointly minimized within a supervised dictionary learning framework, generating a set of quality-oriented MB-LVPs for each single and multiple distortion modality. Given an input stereoscopic image, feature encoding is performed using the learned MB-LVPs as codebooks, resulting in the corresponding monocular and binocular responses. Finally, responses across all the modalities are fused with probabilistic weights which are determined by the modality-specific sparse reconstruction errors, yielding the final monocular and binocular features for quality regression. The superiority of our method has been verified on several SDSI and MDSI databases.

Multiple just-noticeable-difference-based no-reference stereoscopic image quality assessment

Just-noticeable difference (JND) is an important characteristic of the human visual system (HVS), and some established JND models imitating the perception of human eyes already exist. However, their utilization in stereoscopic image quality assessment (SIQA) remains limited. To better simulate how HVS senses 3D images under a no-reference situation, a novel SIQA method based on multiple JND models is proposed in this paper. In our metric, the stereoscopic image pairs are decomposed into multi-scale monocular views and binocular views. Then, texture and edge information of these multi-scale images is extracted. Next, a monocular JND model, a binocular JND model, and a depth JND model are separately applied to the extracted features and the depth map. Finally, these features are synthesized and mapped to objective scores. Through experiment and comparison on public 3D image databases, the proposed method shows a competitive advantage over most state-of-the-art SIQA methods, which indicates that it has a promising prospect in practical applications.

Estimating Depth-Salient Edges and Its Application to Stereoscopic Image Quality Assessment

The human visual system pays attention to salient regions while perceiving an image. When viewing a stereoscopic 3-D (S3D) image, we hypothesize that while most of the contribution to saliency is provided by the 2-D image, a small but significant contribution is provided by the depth component. Further, we claim that only a subset of image edges contribute to depth perception while viewing an S3D image. In this paper, we propose a systematic approach for depth saliency estimation, called salient edges with respect to depth perception (SED) which localizes the depth-salient edges in an S3D image. We demonstrate the utility of SED in full reference stereoscopic image quality assessment. We consider gradient magnitude and inter-gradient maps for predicting structural similarity. A coarse quality map is estimated first by comparing the 2-D saliency and gradient maps of reference and test stereo pairs. We average this quality map to estimate luminance quality and refine this quality map using SED maps for evaluating depth quality. Finally, we combine this luminance and depth quality to obtain an overall stereo image quality. We perform a comprehensive evaluation of our metric on seven publicly available S3D IQA databases. The proposed metric shows competitive performance on all seven databases with state-of-the-art performance on three of them.

No-reference stereoscopic image quality assessment based on hue summation-difference mapping image and binocular joint mutual filtering

The no-reference (NR) quality assessment for stereoscopic images plays a significant role in 3D technology, but it also faces great challenges. In this paper, a novel NR stereo image quality assessment (SIQA) method is proposed. Based on the human visual system, this method mimics the summation and difference channels, which consider the binocular interactive perception property, to process the visual information. Especially, the summation and difference images are calculated by the contrast of hue and luminance in color patches. Meanwhile, considering the interactive filtering between the left and right viewpoints, this method uses the filtered information as the weighting factor to integrate the visual information of the summation and difference channels to form the summation-difference mapping image (SDMI). Then, energy entropy, bivariate generalized Gaussian distribution for the joint distribution of SDMI and the depth map subband coefficients, and the local log-Euclidean multivariate Gaussian descriptor are detected as the feature descriptors. Support vector regression, trained by the features, is utilized to predict the quality of stereoscopic images. Experimental results demonstrate that the proposed algorithm achieves high consistency with subjective assessment on four SIQA databases.

Completely blind image quality assessment via image gray-scale fluctuations and fractal dimension analysis

State-of-the-art no-reference image quality assessment methods usually learn to evaluate image quality by regression from the human subjective scores of a training set. Their dependence on the regression algorithm and human subjective scores may limit the practical application of such methods. In this paper, we propose a completely blind image quality assessment method that is highly unsupervised and training free. We first use a specific image primitive to analyze the image gray-scale fluctuation and use this result as one of the image quality assessment features. The box-counting method is then used to evaluate the image fractal dimension, and the result is used as the other feature. Finally, the two features are combined together, and a formula is introduced to calculate a comprehensive image quality feature, which is used to measure the image quality. Experimental results on four open databases show that the newly proposed method correlates well with the human subjective judgments of diversely distorted images.

Full-reference quality assessment of stereoscopic images by learning binocular visual properties

Stereoscopic imaging technology has been growingly prevalent driven by both the entertainment industry and scientific applications in today's world. But objective quality assessment of stereoscopic images is a challenging task. In this paper, we propose a novel stereoscopic image quality assessment (SIQA) method by jointly considering monocular perception and binocular interaction. As the most significant contribution of this study, binocular perceptual properties of simple and complex cells are considered for full-reference (FR) SIQA. Specifically, the proposed scheme first simulates the receptive fields of simple cells (one class of V1 neurons) using a push-pull combination of receptive fields response, which is used to represent a monocular cue. Further, the receptive fields of complex cells (the other class of V1 neurons) are simulated by using binocular energy response and binocular rivalry response, which are used to represent a binocular cue. Subsequently, various quality-aware features are extracted from the response of area V1 by calculating the self-weighted histogram of the local binary pattern on four types of feature maps of similarity measurement that will change in the presence of distortions. Finally, kernel ridge regression is used to simulate a nonlinear relationship between the quality-aware features and objective quality scores. The performance of our method is evaluated over popular stereoscopic image databases and shown to be competitive with the state-of-the-art FR SIQA algorithms.

Visual Attention Modeling for Stereoscopic Video: A Benchmark and Computational Model

In this paper, we investigate the visual attention modeling for stereoscopic video from the following two aspects. First, we build one large-scale eye tracking database as the benchmark of visual attention modeling for stereoscopic video. The database includes 47 video sequences and their corresponding eye fixation data. Second, we propose a novel computational model of visual attention for stereoscopic video based on Gestalt theory. In the proposed model, we extract the low-level features, including luminance, color, texture, and depth, from discrete cosine transform coefficients, which are used to calculate feature contrast for the spatial saliency computation. The temporal saliency is calculated by the motion contrast from the planar and depth motion features in the stereoscopic video sequences. The final saliency is estimated by fusing the spatial and temporal saliency with uncertainty weighting, which is estimated by the laws of proximity, continuity, and common fate in Gestalt theory. Experimental results show that the proposed method outperforms the state-of-the-art stereoscopic video saliency detection models on our built large-scale eye tracking database and one other database (DML-ITRACK-3D).

Perceptual Depth Quality in Distorted Stereoscopic Images

Subjective and objective measurement of the perceptual quality of depth information in symmetrically and asymmetrically distorted stereoscopic images is a fundamentally important issue in stereoscopic 3D imaging that has not been deeply investigated. Here, we first carry out a subjective test following the traditional absolute category rating protocol widely used in general image quality assessment research. We find this approach problematic, because monocular cues and the spatial quality of images have strong impact on the depth quality scores given by subjects, making it difficult to single out the actual contributions of stereoscopic cues in depth perception. To overcome this problem, we carry out a novel subjective study where depth effect is synthesized at different depth levels before various types and levels of symmetric and asymmetric distortions are applied. Instead of following the traditional approach, we ask subjects to identify and label depth polarizations, and a depth perception difficulty index (DPDI) is developed based on the percentage of correct and incorrect subject judgements. We find this approach highly effective at quantifying depth perception induced by stereo cues and observe a number of interesting effects regarding image content dependency, distortion-type dependence, and the impact of symmetric versus asymmetric distortions. Furthermore, we propose a novel computational model for DPDI prediction. Our results show that the proposed model, without explicitly identifying image distortion types, leads to highly promising DPDI prediction performance. We believe that these are useful steps toward building a comprehensive understanding on 3D quality-of-experience of stereoscopic images.

High-Efficiency 3D Depth Coding Based on Perceptual Quality of Synthesized Video

In 3D video systems, imperfect depth images often induce annoying temporal noise, e.g., flickering, to the synthesized video. However, the quality of synthesized view is usually measured with peak signal-to-noise ratio or mean squared error, which mainly focuses on pixelwise frame-by-frame distortion regardless of the obvious temporal artifacts. In this paper, a novel full reference synthesized video quality metric (SVQM) is proposed to measure the perceptual quality of the synthesized video in 3D video systems. Based on the proposed SVQM, an improved rate-distortion optimization (RDO) algorithm is developed with the target of minimizing the perceptual distortion of synthesized view at given bit rate. Then, the improved RDO algorithm is incorporated into the 3D High Efficiency Video Coding (3D-HEVC) software to improve the 3D depth video coding efficiency. Experimental results show that the proposed SVQM metric has better consistency with human perception on evaluating the synthesized view compared with the state-of-the-art image/video quality assessment algorithms. Meanwhile, this SVQM metric maintains low complexity and easy integration to the current video codec. In addition, the proposed SVQM-based depth coding scheme can achieve approximately 15.27% and 17.63% overall bit rate reduction or 0.42- and 0.46-dB gain in terms of SVQM quality score on average as compared with the latest 3D-HEVC reference model and the state-of-the-art depth coding algorithm, respectively.

Simulating receptive fields of human visual cortex for 3D image quality prediction

Quality assessment of 3D images presents many challenges when attempting to gain better understanding of the human visual system. In this paper, we propose a new 3D image quality prediction approach by simulating receptive fields (RFs) of human visual cortex. To be more specific, we extract the RFs from a complete visual pathway, and calculate their similarity indices between the reference and distorted 3D images. The final quality score is obtained by determining their connections via support vector regression. Experimental results on three 3D image quality assessment databases demonstrate that in comparison with the most relevant existing methods, the devised algorithm achieves high consistency alignment with subjective assessment, especially for asymmetrically distorted stereoscopic images.

Toward a Blind Deep Quality Evaluator for Stereoscopic Images Based on Monocular and Binocular Interactions

During recent years, blind image quality assessment (BIQA) has been intensively studied with different machine learning tools. Existing BIQA metrics, however, do not design for stereoscopic images. We believe this problem can be resolved by separating 3D images and capturing the essential attributes of images via deep neural network. In this paper, we propose a blind deep quality evaluator (DQE) for stereoscopic images (denoted by 3D-DQE) based on monocular and binocular interactions. The key technical steps in the proposed 3D-DQE are to train two separate 2D deep neural networks (2D-DNNs) from 2D monocular images and cyclopean images to model the process of monocular and binocular quality predictions, and combine the measured 2D monocular and cyclopean quality scores using different weighting schemes. Experimental results on four public 3D image quality assessment databases demonstrate that in comparison with the existing methods, the devised algorithm achieves high consistent alignment with subjective assessment.