A full reference quality assessment method with fused monocular and binocular features for stereo images

Aiming to automatically monitor and improve stereoscopic image and video processing systems, stereoscopic image quality assessment approaches are becoming more and more important as 3D technology gains popularity. We propose a full-reference stereoscopic image quality assessment method that incorporate monocular and binocular features based on binocular competition and binocular integration. To start, we create a three-channel RGB fused view by fusing Gabor filter bank responses and disparity maps. Then, using the monocular view and the RGB fusion view, respectively, we extract monocular and binocular features. To alter the local features in the binocular features, we simultaneously estimate the saliency of the RGB fusion image. Finally, the monocular and binocular quality scores are calculated based on the monocular and binocular features, and the quality scores of the stereo image prediction are obtained by fusion. Performance testing in the LIVE 3D IQA database Phase I and Phase II. The results of the proposed method are compared with newer methods. The experimental results show good consistency and robustness.

An objective measurement approach to quantify the perceived distortions of spectacle lenses

The eye's natural aging influences our ability to focus on close objects. Without optical correction, all adults will suffer from blurry close vision starting in their 40s. In effect, different optical corrections are necessary for near and far vision. Current state-of-the-art glasses offer a gradual change of correction across the field of view for any distance-using Progressive Addition Lenses (PALs). However, an inevitable side effect of PALs is geometric distortion, which causes the swim effect, a phenomenon of unstable perception of the environment leading to discomfort for many wearers. Unfortunately, little is known about the relationship between lens distortions and their perceptual effects, that is, between the complex physical distortions on the one hand and their subjective severity on the other. We show that perceived distortion can be measured as a psychophysical scaling function using a VR experiment with accurately simulated PAL distortions. Despite the multi-dimensional space of physical distortions, the measured perception is well represented as a 1D scaling function; distortions are perceived less with negative far correction, suggesting an advantage for short-sighted people. Beyond that, our results successfully demonstrate that psychophysical scaling with ordinal embedding methods can investigate complex perceptual phenomena like lens distortions that affect geometry, stereo, and motion perception. Our approach provides a new perspective on lens design based on modeling visual processing that could be applied beyond distortions. We anticipate that future PAL designs could be improved using our method to minimize subjectively discomforting distortions rather than merely optimizing physical parameters.

On the utility of Colour in shape analysis: An introduction to Colour science via palaeographical case studies

In this article, we explore the use of colour for the analysis of shapes in digital images. We argue that colour can provide unique information that is not available from shape alone, and that familiarity with the interdisciplinary field of colour science is essential for unlocking the potential of colour. Within this perspective, we offer an illustrated overview of the colour-related aspects of image management and processing, perceptual psychology, and cultural studies, using for exemplary purposes case studies focused on computational palaeography. We also discuss the changing roles of colour in society and the sciences, and provide technical solutions for using digital colour effectively, highlighting the impact of human factors. The article concludes with an annotated bibliography. This work is a primer, and its intended readership are scholars and computer scientists unfamiliar with colour science.

Stochastic surprisal: An inferential measurement of free energy in neural networks

This paper conjectures and validates a framework that allows for action during inference in supervised neural networks. Supervised neural networks are constructed with the objective to maximize their performance metric in any given task. This is done by reducing free energy and its associated surprisal during training. However, the bottom-up inference nature of supervised networks is a passive process that renders them fallible to noise. In this paper, we provide a thorough background of supervised neural networks, both generative and discriminative, and discuss their functionality from the perspective of free energy principle. We then provide a framework for introducing action during inference. We introduce a new measurement called stochastic surprisal that is a function of the network, the input, and any possible action. This action can be any one of the outputs that the neural network has learnt, thereby lending stochasticity to the measurement. Stochastic surprisal is validated on two applications: Image Quality Assessment and Recognition under noisy conditions. We show that, while noise characteristics are ignored to make robust recognition, they are analyzed to estimate image quality scores. We apply stochastic surprisal on two applications, three datasets, and as a plug-in on 12 networks. In all, it provides a statistically significant increase among all measures. We conclude by discussing the implications of the proposed stochastic surprisal in other areas of cognitive psychology including expectancy-mismatch and abductive reasoning.

Degraded Reference Image Quality Assessment

In practical media distribution systems, visual content usually undergoes multiple stages of quality degradation along the delivery chain, but the pristine source content is rarely available at most quality monitoring points along the chain to serve as a reference for quality assessment. As a result, full-reference (FR) and reduced-reference (RR) image quality assessment (IQA) methods are generally infeasible. Although no-reference (NR) methods are readily applicable, their performance is often not reliable. On the other hand, intermediate references of degraded quality are often available, e.g., at the input of video transcoders, but how to make the best use of them in proper ways has not been deeply investigated. Here we make one of the first attempts to establish a new paradigm named degraded-reference IQA (DR IQA). Specifically, by using a two-stage distortion pipeline we lay out the architectures of DR IQA and introduce a 6-bit code to denote the choices of configurations. We construct the first large-scale databases dedicated to DR IQA and will make them publicly available. We make novel observations on distortion behavior in multi-stage distortion pipelines by comprehensively analyzing five multiple distortion combinations. Based on these observations, we develop novel DR IQA models and make extensive comparisons with a series of baseline models derived from top-performing FR and NR models. The results suggest that DR IQA may offer significant performance improvement in multiple distortion environments, thereby establishing DR IQA as a valid IQA paradigm that is worth further exploration.

Modification and evaluation of the peripheral contrast sensitivity function models

We propose a series of modifications to the Barten contrast sensitivity function model for peripheral vision based on anatomical and psychophysical studies. These modifications result in a luminance pattern detection model that could quantitatively describe the extent of veridical pattern resolution and the aliasing zone. We evaluated our model against psychophysical measurements in peripheral vision. Our numerical assessment shows that the modified Barten leads to lower estimate errors than its original version.

Contrast discrimination in images of natural scenes

Contrast discrimination determines the threshold contrast required to distinguish between two suprathreshold visual stimuli. It is typically measured using sine-wave gratings. We first present a modification to Barten's semi-mechanistic contrast discrimination model to account for spatial frequency effects and demonstrate how the model can successfully predict visual thresholds obtained from published classical contrast discrimination studies. Contrast discrimination functions are then measured from images of natural scenes, using a psychophysical paradigm based on that employed in our previous study of contrast detection sensitivity. The proposed discrimination model modification is shown to successfully predict discrimination thresholds for structurally very different types of natural image stimuli. A comparison of results shows that, for normal contrast levels in natural scene viewing, contextual contrast detection and discrimination are approximately the same and almost independent of spatial frequency within the range of 1-20 c/deg. At higher frequencies, both sensitivities decrease in magnitude due to optical limitations of the eye. The results are discussed in relation to current image quality models.

Quality Control for the BPG Lossy Compression of Three-Channel Remote Sensing Images

This paper deals with providing the desired quality in the Better Portable Graphics (BPG)-based lossy compression of color and three-channel remote sensing (RS) images. Quality is described by the Mean Deviation Similarity Index (MDSI), which is proven to be one of the best metrics for characterizing compressed image quality due to its high conventional and rank-order correlation with the Mean Opinion Score (MOS) values. The MDSI properties are studied and three main areas of interest are determined. It is shown that quite different quality and compression ratios (CR) can be observed for the same values of the quality parameter Q that controls compression, depending on the compressed image complexity. To provide the desired quality, a modified two-step procedure is proposed and tested. It has a preliminary stage carried out offline (in advance). At this stage, an average rate-distortion curve (MDSI on Q) is obtained and it is available until the moment when a given image has to be compressed. Then, in the first step, an image is compressed using the starting Q determined from the average rate-distortion curve for the desired MDSI. After this, the image is decompressed and the produced MDSI is calculated. In the second step, if necessary, the parameter Q is corrected using the average rate-distortion curve, and the image is compressed with the corrected Q. Such a procedure allows a decrease in the MDSI variance by around one order after two steps compared to variance after the first step. This is important for the MDSI of approximately 0.2–0.25 corresponding to the distortion invisibility threshold. The BPG performance comparison to some other coders is performed and examples of its application to real-life RS images are presented.

Multi-gene Genetic Programming based Predictive Models for Full-reference Image Quality Assessment

Abstract Many objective quality metrics for assessing the visual quality of images have been developed during the last decade. A simple way to fine tune the efficiency of assessment is through permutation and combination of these metrics. The goal of this fusion approach is to take advantage of the metrics utilized and minimize the influence of their drawbacks. In this paper, a symbolic regression technique using an evolutionary algorithm known as multi-gene genetic programming (MGGP) is applied for predicting subject scores of images in datasets using a combination of objective scores of a set of image quality metrics (IQM). By learning from image datasets, the MGGP algorithm can determine appropriate image quality metrics, from 21 metrics utilized, whose objective scores employed as predictors in the symbolic regression model, by optimizing simultaneously two competing objectives of model ‘goodness of fit’ to data and model ‘complexity’. Six large image databases (namely LIVE, CSIQ, TID2008, TID2013, IVC and MDID) that are available in public domain are used for learning and testing the predictive models, according the k-fold-cross-validation and the cross dataset strategies. The proposed approach is compared against state-of-the-art objective image quality assessment approaches. Results of comparison reveal that the proposed approach outperforms other state-of-the-art recently developed fusion approaches.

Combined Full-Reference Image Quality Metrics for Objective Assessment of Multiply Distorted Images

In the recent years, many objective image quality assessment methods have been proposed by different researchers, leading to a significant increase in their correlation with subjective quality evaluations. Although many recently proposed image quality assessment methods, particularly full-reference metrics, are in some cases highly correlated with the perception of individual distortions, there is still a need for their verification and adjustment for the case when images are affected by multiple distortions. Since one of the possible approaches is the application of combined metrics, their analysis and optimization are discussed in this paper. Two approaches to metrics’ combination have been analyzed that are based on the weighted product and the proposed weighted sum with additional exponential weights. The validation of the proposed approach, carried out using four currently available image datasets, containing multiply distorted images together with the gathered subjective quality scores, indicates a meaningful increase of correlations of the optimized combined metrics with subjective opinions for all datasets.

Salient Region Guided Blind Image Sharpness Assessment

Salient regions provide important cues for scene understanding to the human vision system. However, whether the detected salient regions are helpful in image blur estimation is unknown. In this study, a salient region guided blind image sharpness assessment (BISA) framework is proposed, and the effect of the detected salient regions on the BISA performance is investigated. Specifically, three salient region detection (SRD) methods and ten BISA models are jointly explored, during which the output saliency maps from SRD methods are re-organized as the input of BISA models. Consequently, the change in BISA metric values can be quantified and then directly related to the difference in BISA model inputs. Finally, experiments are conducted on three Gaussian blurring image databases, and the BISA prediction performance is evaluated. The comparison results indicate that salient region input can help achieve a close and sometimes superior performance to a BISA model over the whole image input. When using the center region input as the baseline, the detected salient regions from the saliency optimization from robust background detection (SORBD) method lead to consistently better score prediction, regardless of the BISA model. Based on the proposed hybrid framework, this study reveals that saliency detection benefits image blur estimation, while how to properly incorporate SRD methods and BISA models to improve the score prediction will be explored in our future work.

Image Quality Assessment without Reference by Combining Deep Learning-Based Features and Viewing Distance

An abundance of objective image quality metrics have been introduced in the literature. One important essential aspect that perceived image quality is dependent on is the viewing distance from the observer to the image. We introduce in this study a novel image quality metric able to estimate the quality of a given image without reference for different viewing distances between the image and the observer. We first select relevant patches from the image using saliency information. For each patch, a feature vector is extracted from a convolutional neural network model and concatenated at the viewing distance, for which the quality is predicted. The resulting vector is fed to fully connected layers to predict subjective scores for the considered viewing distance. The proposed method was evaluated using the Colourlab Image Database: Image Quality and Viewing Distance-changed Image Database. Both databases provide subjective scores at two different viewing distances. In the Colourlab Image Database: Image Quality we obtain a Pearson correlation of 0.87 at both 50 cm and 100 cm viewing distances, while in the Viewing Distance-changed Image Database we obtained a Pearson correlation of 0.93 and 0.94 at viewing distance of four and six times the image height. The results show the efficiency of our method and its generalization ability.

Selection of Lee Filter Window Size Based on Despeckling Efficiency Prediction for Sentinel SAR Images

Radar imaging has many advantages. Meanwhile, SAR images suffer from a noise-like phenomenon called speckle. Many despeckling methods have been proposed to date but there is still no common opinion as to what the best filter is and/or what are its parameters (window or block size, thresholds, etc.). The local statistic Lee filter is one of the most popular and best-known despeckling techniques in radar image processing. Using this filter and Sentinel-1 images as a case study, we show how filter parameters, namely scanning window size, can be selected for a given image based on filter efficiency prediction. Such a prediction can be carried out using a set of input parameters that can be easily and quickly calculated and employing a trained neural network that allows determining one or several criteria of filtering efficiency with high accuracy. The statistical analysis of the obtained results is carried out. This characterizes improvements due to the adaptive selection of the filter window size, both potential and based on prediction. We also analyzed what happens if, due to prediction errors, erroneous decisions are undertaken. Examples for simulated and real-life images are presented.

Challenges and opportunities for artificial intelligence in oncological imaging

Imaging plays a key role in oncology, including the diagnosis and detection of cancer, determining clinical management, assessing treatment response, and complications of treatment or disease. The current use of clinical oncology is predominantly qualitative in nature with some relatively crude size-based measurements of tumours for assessment of disease progression or treatment response; however, it is increasingly understood that there may be significantly more information about oncological disease that can be obtained from imaging that is not currently utilized. Artificial intelligence (AI) has the potential to harness quantitative techniques to improve oncological imaging. These may include improving the efficiency or accuracy of traditional roles of imaging such as diagnosis or detection. These may also include new roles for imaging such as risk-stratifying patients for different types of therapy or determining biological tumour subtypes. This review article outlines several major areas in oncological imaging where there may be opportunities for AI technology. These include (1) screening and detection of cancer, (2) diagnosis and risk stratification, (3) tumour segmentation, (4) precision oncology, and (5) predicting prognosis and assessing treatment response. This review will also address some of the potential barriers to AI research in oncological imaging.

SSIM Compliant Modeling Framework With Denoising and Deblurring Applications

In image processing, it is well known that mean square error criteria is perceptually inadequate. Consequently, image quality assessment (IQA) has emerged as a new branch to overcome this issue, and this has led to the discovery of one of the most popular perceptual measures, namely, the structural similarity index (SSIM). This measure is mathematically simple, yet powerful enough to express the quality of an image. Therefore, it is natural to deploy SSIM in model based applications, such as denoising, restoration, classification, etc. However, the non-convex nature of this measure makes this task difficult. Our attempt in this work is to discuss problems associated with its convex program and take remedial action in the process of obtaining a generalized convex framework. The obtained framework has been seen as a component of an alternative learning scheme for the case of a regularized linear model. Subsequently, we develop a relevant dictionary learning module as a part of alternative learning. This alternative learning scheme with sparsity prior is finally used in denoising and deblurring applications. To further boost the performance, an iterative scheme is developed based on the statistical nature of added noise. Experiments on image denoising and deblurring validate the effectiveness of the proposed scheme. Furthermore, it has been shown that the proposed framework achieves highly competitive performance with respect to other schemes in literature and performs better in natural images in terms of SSIM and visual inspection.

High quality and fast compressed sensing MRI reconstruction via edge-enhanced dual discriminator generative adversarial network

Generative adversarial networks (GAN) are widely used for fast compressed sensing magnetic resonance imaging (CSMRI) reconstruction. However, most existing methods are difficult to make an effective trade-off between abstract global high-level features and edge features. It easily causes problems, such as significant remaining aliasing artifacts and clearly over-smoothed reconstruction details. To tackle these issues, we propose a novel edge-enhanced dual discriminator generative adversarial network architecture called EDDGAN for CSMRI reconstruction with high quality. In this model, we extract effective edge features by fusing edge information from different depths. Then, leveraging the relationship between abstract global high-level features and edge features, a three-player game is introduced to control the hallucination of details and stabilize the training process. The resulting EDDGAN can offer more focus on edge restoration and de-aliasing. Extensive experimental results demonstrate that our method consistently outperforms state-of-the-art methods and obtains reconstructed images with rich edge details. In addition, our method also shows remarkable generalization, and its time consumption for each 256 × 256 image reconstruction is approximately 8.39 ms.

Screen Content Video Quality Assessment: Subjective and Objective Study

In this paper, we make the first attempt to study the subjective and objective quality assessment for the screen content videos (SCVs). For that, we construct the first large-scale video quality assessment (VQA) database specifically for the SCVs, called the screen content video database (SCVD). This SCVD provides 16 reference SCVs, 800 distorted SCVs, and their corresponding subjective scores, and it is made publicly available for research usage. The distorted SCVs are generated from each reference SCV with 10 distortion types and 5 degradation levels for each distortion type. Each distorted SCV is rated by at least 32 subjects in the subjective test. Furthermore, we propose the first full-reference VQA model for the SCVs, called the spatiotemporal Gabor feature tensor-based model (SGFTM), to objectively evaluate the perceptual quality of the distorted SCVs. This is motivated by the observation that 3D-Gabor filter can well stimulate the visual functions of the human visual system (HVS) on perceiving videos, being more sensitive to the edge and motion information that are often-encountered in the SCVs. Specifically, the proposed SGFTM exploits 3D-Gabor filter to individually extract the spatiotemporal Gabor feature tensors from the reference and distorted SCVs, followed by measuring their similarities and later combining them together through the developed spatiotemporal feature tensor pooling strategy to obtain the final SGFTM score. Experimental results on SCVD have shown that the proposed SGFTM yields a high consistency on the subjective perception of SCV quality and consistently outperforms multiple classical and state-of-the-art image/video quality assessment models.

Full-Reference Quality Metric Based on Neural Network to Assess the Visual Quality of Remote Sensing Images

Remote sensing images are subject to different types of degradations. The visual quality of such images is important because their visual inspection and analysis are still widely used in practice. To characterize the visual quality of remote sensing images, the use of specialized visual quality metrics is desired. Although the attempts to create such metrics are limited, there is a great number of visual quality metrics designed for other applications. Our idea is that some of these metrics can be employed in remote sensing under the condition that those metrics have been designed for the same distortion types. Thus, image databases that contain images with types of distortions that are of interest should be looked for. It has been checked what known visual quality metrics perform well for images with such degradations and an opportunity to design neural network-based combined metrics with improved performance has been studied. It is shown that for such combined metrics, their Spearman correlation coefficient with mean opinion score exceeds 0.97 for subsets of images in the Tampere Image Database (TID2013). Since different types of elementary metric pre-processing and neural network design have been considered, it has been demonstrated that it is enough to have two hidden layers and about twenty inputs. Examples of using known and designed visual quality metrics in remote sensing are presented.

No-reference quality assessment for image-based assessment of economically important tropical woods

Image Quality Assessment (IQA) is essential for the accuracy of systems for automatic recognition of tree species for wood samples. In this study, a No-Reference IQA (NR-IQA), wood NR-IQA (WNR-IQA) metric was proposed to assess the quality of wood images. Support Vector Regression (SVR) was trained using Generalized Gaussian Distribution (GGD) and Asymmetric Generalized Gaussian Distribution (AGGD) features, which were measured for wood images. Meanwhile, the Mean Opinion Score (MOS) was obtained from the subjective evaluation. This was followed by a comparison between the proposed IQA metric, WNR-IQA, and three established NR-IQA metrics, namely Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE), deepIQA, Deep Bilinear Convolutional Neural Networks (DB-CNN), and five Full Reference-IQA (FR-IQA) metrics known as MSSIM, SSIM, FSIM, IWSSIM, and GMSD. The proposed WNR-IQA metric, BRISQUE, deepIQA, DB-CNN, and FR-IQAs were then compared with MOS values to evaluate the performance of the automatic IQA metrics. As a result, the WNR-IQA metric exhibited a higher performance compared to BRISQUE, deepIQA, DB-CNN, and FR-IQA metrics. Highest quality images may not be routinely available due to logistic factors, such as dust, poor illumination, and hot environment present in the timber industry. Moreover, motion blur could occur due to the relative motion between the camera and the wood slice. Therefore, the advantage of WNR-IQA could be seen from its independency from a “perfect” reference image for the image quality evaluation.

The Performance of Quality Metrics in Assessing Error-Concealed Video Quality

In highly-interactive video streaming applications such as video conferencing, tele-presence, or tele-operation, retransmission is typically not used, due to the tight deadline of the application. In such cases, the lost or erroneous data must be concealed. While various error concealment techniques exist, there is no defined rule to compare their perceived quality. In this paper, the performance of 16 existing image and video quality metrics (PSNR, SSIM, VQM, etc.) evaluating errorconcealed video quality is studied. The encoded video is subjected to packet loss and the loss is concealed using various error concealment techniques. We show that the subjective quality of the video cannot be necessarily predicted from the visual quality of the error-concealed frame alone. We then apply the metrics to the error-concealed images/videos and evaluate their success in predicting the scores reported by human subjects. The errorconcealed videos are judged by image quality metrics applied on the lossy frame, or by video quality metrics applied on the video clip containing that lossy frame; this way, the impact of error propagation is also considered by the objective metrics. The measurement and comparison of the results show that, mostly though not always, measuring the objective quality of the video is a better way to judge the error concealment performance. Moreover, our experiments show that when the objective quality metrics are used for the assessment of the performance of an error concealment technique, they do not behave as they would for general quality assessment. In fact, some newly developed metrics show the correct decision only about 60% of the time, leading to an unacceptable error rate of as much as 40%. Our analysis shows which specific quality metrics are relatively more suitable for error-concealed videos.

Magnetic Resonance Image Quality Assessment by Using Non-Maximum Suppression and Entropy Analysis

An investigation of diseases using magnetic resonance (MR) imaging requires automatic image quality assessment methods able to exclude low-quality scans. Such methods can be also employed for an optimization of parameters of imaging systems or evaluation of image processing algorithms. Therefore, in this paper, a novel blind image quality assessment (BIQA) method for the evaluation of MR images is introduced. It is observed that the result of filtering using non-maximum suppression (NMS) strongly depends on the perceptual quality of an input image. Hence, in the method, the image is first processed by the NMS with various levels of acceptable local intensity difference. Then, the quality is efficiently expressed by the entropy of a sequence of extrema numbers obtained with the thresholded NMS. The proposed BIQA approach is compared with ten state-of-the-art techniques on a dataset containing MR images and subjective scores provided by 31 experienced radiologists. The Pearson, Spearman, Kendall correlation coefficients and root mean square error for the method assessing images in the dataset were 0.6741, 0.3540, 0.2428, and 0.5375, respectively. The extensive experimental evaluation of the BIQA methods reveals that the introduced measure outperforms related techniques by a large margin as it correlates better with human scores.

Universal Measure for Medical Image Quality Evaluation Based on Gradient Approach

In this paper, a new universal measure of medical images quality is proposed. The measure is based on the analysis of the image by using gradient methods. The number of isolated peaks in the examined image, as a function of the threshold value, is the basis of the assessment of the image quality. It turns out that for higher quality images the curvature of the graph of the said function has a higher value for lower threshold values. On the basis of the observed property, a new method of no-reference image quality assessment has been created. The experimental verification confirmed the method efficiency. The correlation between the arrangement depending on the image quality done by an expert and by using the proposed method is equal to 0.74. This means that the proposed method gives a correlation of higher than the best methods described in the literature. The proposed measure is useful to maximize the image quality while minimizing the time of medical examination.

Learning No-Reference Quality Assessment of Multiply and Singly Distorted Images with Big Data

Previous research on no-reference (NR) quality assessment of multiply-distorted images focused mainly on three distortion types (white noise, Gaussian blur, and JPEG compression), while in practice images can be contaminated by many other common distortions due to the various stages of processing. Although MUSIQUE (MUltiply-and Singly-distorted Image QUality Estimator) Zhang et al., TIP 2018 is a successful NR algorithm, this approach is still limited to the three distortion types. In this paper, we extend MUSIQUE to MUSIQUE-II to blindly assess the quality of images corrupted by five distortion types (white noise, Gaussian blur, JPEG compression, JPEG2000 compression, and contrast change) and their combinations. The proposed MUSIQUE-II algorithm builds upon the classification and parameter-estimation framework of its predecessor by using more advanced models and a more comprehensive set of distortion-sensitive features. Specifically, MUSIQUE-II relies on a three-layer classification model to identify 19 distortion types. To predict the five distortion parameter values, MUSIQUE-II extracts an additional 14 contrast features and employs a multi-layer probability-weighting rule. Finally, MUSIQUE-II employs a new most-apparent-distortion strategy to adaptively combine five quality scores based on outputs of three classification models. Experimental results tested on three multiply-distorted and six singly-distorted image quality databases show that MUSIQUE-II yields not only a substantial improvement in quality predictive performance as compared with its predecessor, but also highly competitive performance relative to other state-of-the-art FR/NR IQA algorithms.

An Evaluation Framework for Spectral Filter Array Cameras to Optimize Skin Diagnosis

Comparing and selecting an adequate spectral filter array (SFA) camera is application-specific and usually requires extensive prior measurements. An evaluation framework for SFA cameras is proposed and three cameras are tested in the context of skin analysis. The proposed framework does not require application-specific measurements and spectral sensitivities together with the number of bands are the main focus. An optical model of skin is used to generate a specialized training set to improve spectral reconstruction. The quantitative comparison of the cameras is based on reconstruction of measured skin spectra, colorimetric accuracy, and oxygenation level estimation differences. Specific spectral sensitivity shapes influence the results directly and a 9-channel camera performed best regarding the spectral reconstruction metrics. Sensitivities at key wavelengths influence the performance of oxygenation level estimation the strongest. The proposed framework allows to compare spectral filter array cameras and can guide their application-specific development.

Visual Quality Assessment for Super-Resolved Images: Database and Method

Image super-resolution (SR) has been an active research problem which has recently received renewed interest due to the introduction of new technologies such as deep learning. However, the lack of suitable criteria to evaluate the SR performance has hindered technology development. In this paper, we fill a gap in the literature by providing the first publicly available database as well as a new image quality assessment (IQA) method specifically designed for assessing the visual quality of super-resolved images (SRIs). In constructing the quality assessment database for SRIs (QADS), we carefully selected 20 reference images and created 980 SRIs using 21 image SR methods. Mean opinion score (MOS) for these SRIs is collected through 100 individuals participating in a suitably designed psychovisual experiment. Extensive numerical and statistical analysis is performed to show that the MOS of QADS has excellent suitability and reliability. The psychovisual experiment has led to the discovery that, unlike distortions encountered in other IQA databases, artifacts of the SRIs degenerate the image structure as well as the image texture. Moreover, the structural and textural degenerations have distinctive perceptual properties. Based on these insights, we propose a novel method to assess the visual quality of SRIs by separately considering the structural and textural components of images. Observing that textural degenerations are mainly attributed to dissimilar texture or checkerboard artifacts, we propose to measure the changes of textural distributions. We also observe that structural degenerations appear as blurring and jaggies artifacts in SRIs and develop separate similarity measures for different types of structural degenerations. A new pooling mechanism is then used to fuse the different similarities together to give the final quality score for an SRI. The experiments conducted on the QADS demonstrate that our method significantly outperforms the classical as well as current state-of-the-art IQA methods.

No-Reference Image Quality Assessment with Local Gradient Orientations

Image processing methods often introduce distortions, which affect the way an image is subjectively perceived by a human observer. To avoid inconvenient subjective tests in cases in which reference images are not available, it is desirable to develop an automatic no-reference image quality assessment (NR-IQA) technique. In this paper, a novel NR-IQA technique is proposed in which the distributions of local gradient orientations in image regions of different sizes are used to characterize an image. To evaluate the objective quality of an image, its luminance and chrominance channels are processed, as well as their high-order derivatives. Finally, statistics of used perceptual features are mapped to subjective scores by the support vector regression (SVR) technique. The extensive experimental evaluation on six popular IQA benchmark datasets reveals that the proposed technique is highly correlated with subjective scores and outperforms related state-of-the-art hand-crafted and deep learning approaches.

On the Application LBP Texture Descriptors and Its Variants for No-Reference Image Quality Assessment

Automatic assessing the quality of an image is a critical problem for a wide range of applications in the fields of computer vision and image processing. For example, many computer vision applications, such as biometric identification, content retrieval, and object recognition, rely on input images with a specific range of quality. Therefore, an effort has been made to develop image quality assessment (IQA) methods that are able to automatically estimate quality. Among the possible IQA approaches, No-Reference IQA (NR-IQA) methods are of fundamental interest, since they can be used in most real-time multimedia applications. NR-IQA are capable of assessing the quality of an image without using the reference (or pristine) image. In this paper, we investigate the use of texture descriptors in the design of NR-IQA methods. The premise is that visible impairments alter the statistics of texture descriptors, making it possible to estimate quality. To investigate if this premise is valid, we analyze the use of a set of state-of-the-art Local Binary Patterns (LBP) texture descriptors in IQA methods. Particularly, we present a comprehensive review with a detailed description of the considered methods. Additionally, we propose a framework for using texture descriptors in NR-IQA methods. Our experimental results indicate that, although not all texture descriptors are suitable for NR-IQA, many can be used with this purpose achieving a good accuracy performance with the advantage of a low computational complexity.

GPU Acceleration of the Most Apparent Distortion Image Quality Assessment Algorithm

The primary function of multimedia systems is to seamlessly transform and display content to users while maintaining the perception of acceptable quality. For images and videos, perceptual quality assessment algorithms play an important role in determining what is acceptable quality and what is unacceptable from a human visual perspective. As modern image quality assessment (IQA) algorithms gain widespread adoption, it is important to achieve a balance between their computational efficiency and their quality prediction accuracy. One way to improve computational performance to meet real-time constraints is to use simplistic models of visual perception, but such an approach has a serious drawback in terms of poor-quality predictions and limited robustness to changing distortions and viewing conditions. In this paper, we investigate the advantages and potential bottlenecks of implementing a best-in-class IQA algorithm, Most Apparent Distortion, on graphics processing units (GPUs). Our results suggest that an understanding of the GPU and CPU architectures, combined with detailed knowledge of the IQA algorithm, can lead to non-trivial speedups without compromising prediction accuracy. A single-GPU and a multi-GPU implementation showed a 24× and a 33× speedup, respectively, over the baseline CPU implementation. A bottleneck analysis revealed the kernels with the highest runtimes, and a microarchitectural analysis illustrated the underlying reasons for the high runtimes of these kernels. Programs written with optimizations such as blocking that map well to CPU memory hierarchies do not map well to the GPU’s memory hierarchy. While compute unified device architecture (CUDA) is convenient to use and is powerful in facilitating general purpose GPU (GPGPU) programming, knowledge of how a program interacts with the underlying hardware is essential for understanding performance bottlenecks and resolving them.

SPSIM: A Superpixel-Based Similarity Index for Full-Reference Image Quality Assessment

Full-reference image quality assessment algorithms usually perform comparisons of features extracted from square patches. These patches do not have any visual meanings. On the contrary, a superpixel is a set of image pixels that share similar visual characteristics and is thus perceptually meaningful. Features from superpixels may improve the performance of image quality assessment. Inspired by this, we propose a new superpixel-based similarity index by extracting perceptually meaningful features and revising similarity measures. The proposed method evaluates image quality on the basis of three measurements, namely, superpixel luminance similarity, superpixel chrominance similarity, and pixel gradient similarity. The first two measurements assess the overall visual impression on local images. The third measurement quantifies structural variations. The impact of superpixel-based regional gradient consistency on image quality is also analyzed. Distorted images showing high regional gradient consistency with the corresponding reference images are visually appreciated. Therefore, the three measurements are further revised by incorporating the regional gradient consistency into their computations. A weighting function that indicates superpixel-based texture complexity is utilized in the pooling stage to obtain the final quality score. Experiments on several benchmark databases demonstrate that the proposed method is competitive with the state-of-the-art metrics.

Opinion-Unaware Blind Quality Assessment of Multiply and Singly Distorted Images via Distortion Parameter Estimation

Over the past couple of decades, numerous image quality assessment (IQA) algorithms have been developed to estimate the quality of images that contain a single type of distortion. Although in practice, images can be contaminated by multiple distortions, previous research on quality assessment of multiply-distorted images is very limited. In this paper, we propose an efficient algorithm to blindly assess the quality of both multiply and singly distorted images based on predicting the distortion parameters using a bag of natural scene statistics (NSS) features. Our method, called MUltiply- and Singlydistorted Image QUality Estimator (MUSIQUE), operates via three main stages. In the first stage, a two-layer classification model is employed to identify the distortion types (i.e., Gaussian blur, JPEG compression, and white noise) that may exist in an image. In the second stage, specific regression models are employed to predict the three distortion parameters (i.e., σG for Gaussian blur, Q for JPEG compression, and σN for white noise) by learning the different NSS features for different distortion types and combinations. In the final stage, the three estimated distortion parameter values are mapped and combined into an overall quality estimate based on quality-mapping curves and the most-apparent-distortion strategy. Experimental results tested on three multiply-distorted and seven singly-distorted image quality databases demonstrate that the proposed MUSIQUE algorithm can achieve better/competitive performance as compared to other state-of-the-art FR/NR IQA algorithms.

Quality Assessment of Screen Content Images via Convolutional-Neural-Network-Based Synthetic/Natural Segmentation

The recent popularity of remote desktop software and live streaming of composited video has given rise to a growing number of applications which make use of so-called screen content images that contain a mixture of text, graphics, and photographic imagery. Automatic quality assessment (QA) of screen-content images is necessary to enable tasks such as quality monitoring, parameter adaptation, and other optimizations. Although QA of natural images has been heavily researched over the last several decades, QA of screen content images is a relatively new topic. In this paper, we present a QA algorithm, called convolutional neural network (CNN) based screen content image quality estimator (CNN-SQE), which operates via a fuzzy classification of screen content images into plain-text, computergraphics/ cartoons, and natural-image regions. The first two classes are considered to contain synthetic content (text/graphics), and the latter two classes are considered to contain naturalistic content (graphics/photographs), where the overlap of the classes allows the computer graphics/cartoons segments to be analyzed by both text-based and natural-image-based features. We present a CNN-based approach for the classification, an edge-structurebased quality degradation model, and a region-size-adaptive quality-fusion strategy. As we will demonstrate, the proposed CNN-SQE algorithm can achieve better/competitive performance as compared with other state-of-the-art QA algorithms.

A Detail-Based Method for Linear Full Reference Image Quality Prediction

In this paper, a novel full Reference method is proposed for image quality assessment, using the combination of two separate metrics to measure the perceptually distinct impact of detail losses and of spurious details. To this purpose, the gradient of the impaired image is locally decomposed as a predicted version of the original gradient, plus a gradient residual. It is assumed that the detail attenuation identifies the detail loss, whereas the gradient residuals describe the spurious details. It turns out that the perceptual impact of detail losses is roughly linear with the loss of the positional Fisher information, while the perceptual impact of the spurious details is roughly proportional to a logarithmic measure of the signal to residual ratio. The affine combination of these two metrics forms a new index strongly correlated with the empirical differential mean opinion score (DMOS) for a significant class of image impairments, as verified for three independent popular databases. The method allowed alignment and merging of DMOS data coming from these different databases to a common DMOS scale by affine transformations. Unexpectedly, the DMOS scale setting is possible by the analysis of a single image affected by additive noise.

Binocular rivalry after right-hemisphere stroke: Effects of attention impairment on perceptual dominance patterns

Binocular rivalry is when perception fluctuates while the stimuli, consisting of different images presented to each eye, remain unchanged. The fluctuation rate and predominance ratio of these images are regarded as information source for understanding properties of consciousness and perception. We administered a binocular rivalry task to 26 right-hemisphere stroke patients and 26 healthy control participants, using stimuli such as simple Gabor anaglyphs. Each single Gabor image was of unequal spatial frequency compared to its counterpart, allowing assessment of the effect of relative spatial frequency on rivalry predominance. Results revealed that patients had significantly decreased alternation rate compared to healthy controls, with severity of patients' attention impairment predicting alternation rates. The patient group had higher predominance ratio for high compared to low relative spatial frequency stimuli consistent with the hypothesis that damage to the right hemisphere may disrupt processing of relatively low spatial frequencies. Degree of attention impairment also predicted the effect of relative spatial frequencies. Lastly, both groups showed increased predominance rates in the right eye compared to the left eye. This right eye dominance was more pronounced in patients than controls, suggesting that right hemisphere stroke may additionally affect eye predominance ratios.

Revealing information by averaging

We present a method for hiding images in synthetic videos and reveal them by temporal averaging. The main challenge is to develop a visual masking method that hides the input image both spatially and temporally. Our masking approach consists of temporal and spatial pixel by pixel variations of the frequency band coefficients representing the image to be hidden. These variations ensure that the target image remains invisible both in the spatial and the temporal domains. In addition, by applying a temporal masking function derived from a dither matrix, we allow the video to carry a visible message that is different from the hidden image. The image hidden in the video can be revealed by software averaging, or with a camera, by long-exposure photography. The presented work may find applications in the secure transmission of digital information.

Sparse Representation-Based Image Quality Index With Adaptive Sub-Dictionaries

Distortions cause structural changes in digital images, leading to degraded visual quality. Dictionary-based sparse representation has been widely studied recently due to its ability to extract inherent image structures. Meantime, it can extract image features with slightly higher level semantics. Intuitively, sparse representation can be used for image quality assessment, because visible distortions can cause significant changes to the sparse features. In this paper, a new sparse representation-based image quality assessment model is proposed based on the construction of adaptive sub-dictionaries. An overcomplete dictionary trained from natural images is employed to capture the structure changes between the reference and distorted images by sparse feature extraction via adaptive sub-dictionary selection. Based on the observation that image sparse features are invariant to weak degradations and the perceived image quality is generally influenced by diverse issues, three auxiliary quality features are added, including gradient, color, and luminance information. The proposed method is not sensitive to training images, so a universal dictionary can be adopted for quality evaluation. Extensive experiments on five public image quality databases demonstrate that the proposed method produces the state-of-the-art results, and it delivers consistently well performances when tested in different image quality databases.

Full-Reference Image Quality Assessment with Linear Combination of Genetically Selected Quality Measures

Information carried by an image can be distorted due to different image processing steps introduced by different electronic means of storage and communication. Therefore, development of algorithms which can automatically assess a quality of the image in a way that is consistent with human evaluation is important. In this paper, an approach to image quality assessment (IQA) is proposed in which the quality of a given image is evaluated jointly by several IQA approaches. At first, in order to obtain such joint models, an optimisation problem of IQA measures aggregation is defined, where a weighted sum of their outputs, i.e., objective scores, is used as the aggregation operator. Then, the weight of each measure is considered as a decision variable in a problem of minimisation of root mean square error between obtained objective scores and subjective scores. Subjective scores reflect ground-truth and involve evaluation of images by human observers. The optimisation problem is solved using a genetic algorithm, which also selects suitable measures used in aggregation. Obtained multimeasures are evaluated on four largest widely used image benchmarks and compared against state-of-the-art full-reference IQA approaches. Results of comparison reveal that the proposed approach outperforms other competing measures.

Feature-Linking Model for Image Enhancement

Inspired by gamma-band oscillations and other neurobiological discoveries, neural networks research shifts the emphasis toward temporal coding, which uses explicit times at which spikes occur as an essential dimension in neural representations. We present a feature-linking model (FLM) that uses the timing of spikes to encode information. The first spiking time of FLM is applied to image enhancement, and the processing mechanisms are consistent with the human visual system. The enhancement algorithm achieves boosting the details while preserving the information of the input image. Experiments are conducted to demonstrate the effectiveness of the proposed method. Results show that the proposed method is effective.

Evaluation of C-arm CT metal artifact reduction algorithm during intra-aneurysmal coil embolization: Assessment of brain parenchyma, stents and flow-diverters

PURPOSE

Flat panel C-arm CT images acquired in the interventional suite provide valuable information regarding brain parenchyma, vasculature, and device status during the procedure. However, these images often suffer from severe streak artifacts due to the presence of metallic objects such as coils. These artifacts limit the capability to make diagnostic inferences and thus need to be reduced for better image interpretation. The main purpose of this paper is to systematically evaluate the accuracy of one such C-arm CT based metal artifact reduction (MAR) algorithm and to demonstrate its usage in both stent and flow diverter assisted coil embolization procedures.

METHODS

C-arm CT images routinely acquired in 24 patients during coil embolization procedure (stent-assisted (12) and flow-diverter assisted (12)) were included in this study in a retrospective fashion. These images were reconstructed without and with MAR algorithm on an offline workstation and compared using quantitative image analysis metrics. This analysis was carried out to assess the improvements in both brain parenchyma and device visibility with MAR algorithm. Further, ground truth reference images from phantom experiments and clinical data were used for accurate assessment.

RESULTS

Quantitative image analysis of brain parenchyma showed uniform distribution of grayscale values and reduced image noise after MAR correction. The line profile plot analysis of device profile in both phantom and clinical data demonstrated improved device visibility with MAR correction.

CONCLUSIONS

MAR algorithm successfully reduced streak artifacts from coil embolization in all cases, thus allowing more accurate assessment of devices and adjacent brain parenchyma.

Learning Receptive Fields and Quality Lookups for Blind Quality Assessment of Stereoscopic Images

Blind quality assessment of 3D images encounters more new challenges than its 2D counterparts. In this paper, we propose a blind quality assessment for stereoscopic images by learning the characteristics of receptive fields (RFs) from perspective of dictionary learning, and constructing quality lookups to replace human opinion scores without performance loss. The important feature of the proposed method is that we do not need a large set of samples of distorted stereoscopic images and the corresponding human opinion scores to learn a regression model. To be more specific, in the training phase, we learn local RFs (LRFs) and global RFs (GRFs) from the reference and distorted stereoscopic images, respectively, and construct their corresponding local quality lookups (LQLs) and global quality lookups (GQLs). In the testing phase, blind quality pooling can be easily achieved by searching optimal GRF and LRF indexes from the learnt LQLs and GQLs, and the quality score is obtained by combining the LRF and GRF indexes together. Experimental results on three publicly 3D image quality assessment databases demonstrate that in comparison with the existing methods, the devised algorithm achieves high consistent alignment with subjective assessment.

No-Reference Image Blur Assessment Based on Discrete Orthogonal Moments

Blur is a key determinant in the perception of image quality. Generally, blur causes spread of edges, which leads to shape changes in images. Discrete orthogonal moments have been widely studied as effective shape descriptors. Intuitively, blur can be represented using discrete moments since noticeable blur affects the magnitudes of moments of an image. With this consideration, this paper presents a blind image blur evaluation algorithm based on discrete Tchebichef moments. The gradient of a blurred image is first computed to account for the shape, which is more effective for blur representation. Then the gradient image is divided into equal-size blocks and the Tchebichef moments are calculated to characterize image shape. The energy of a block is computed as the sum of squared non-DC moment values. Finally, the proposed image blur score is defined as the variance-normalized moment energy, which is computed with the guidance of a visual saliency model to adapt to the characteristic of human visual system. The performance of the proposed method is evaluated on four public image quality databases. The experimental results demonstrate that our method can produce blur scores highly consistent with subjective evaluations. It also outperforms the state-of-the-art image blur metrics and several general-purpose no-reference quality metrics.