Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation

TCDM: Transformational Complexity Based Distortion Metric for Perceptual Point Cloud Quality Assessment

The goal of objective point cloud quality assessment (PCQA) research is to develop quantitative metrics that measure point cloud quality in a perceptually consistent manner. Merging the research of cognitive science and intuition of the human visual system (HVS), in this article, we evaluate the point cloud quality by measuring the complexity of transforming the distorted point cloud back to its reference, which in practice can be approximated by the code length of one point cloud when the other is given. For this purpose, we first make space segmentation for the reference and distorted point clouds based on a 3D Voronoi diagram to obtain a series of local patch pairs. Next, inspired by the predictive coding theory, we utilize a space-aware vector autoregressive (SA-VAR) model to encode the geometry and color channels of each reference patch with and without the distorted patch, respectively. Assuming that the residual errors follow the multi-variate Gaussian distributions, the self-complexity of the reference and transformational complexity between the reference and distorted samples are computed using covariance matrices. Additionally, the prediction terms generated by SA-VAR are introduced as one auxiliary feature to promote the final quality prediction. The effectiveness of the proposed transformational complexity based distortion metric (TCDM) is evaluated through extensive experiments conducted on five public point cloud quality assessment databases. The results demonstrate that TCDM achieves state-of-the-art (SOTA) performance, and further analysis confirms its robustness in various scenarios.

Data Acquisition and Preparation for Dual-Reference Deep Learning of Image Super-Resolution

The performance of deep learning based image super-resolution (SR) methods depend on how accurately the paired low and high resolution images for training characterize the sampling process of real cameras. Low and high resolution (LR  ∼  HR) image pairs synthesized by degradation models (e.g., bicubic downsampling) deviate from those in reality; thus the synthetically-trained DCNN SR models work disappointingly when being applied to real-world images. To address this issue, we propose a novel data acquisition process to shoot a large set of LR  ∼  HR image pairs using real cameras. The images are displayed on an ultra-high quality screen and captured at different resolutions. The resulting LR  ∼  HR image pairs can be aligned at very high sub-pixel precision by a novel spatial-frequency dual-domain registration method, and hence they provide more appropriate training data for the learning task of super-resolution. Moreover, the captured HR image and the original digital image offer dual references to strengthen supervised learning. Experimental results show that training a super-resolution DCNN by our LR  ∼  HR dataset achieves higher image quality than training it by other datasets in the literature. Moreover, the proposed screen-capturing data collection process can be automated; it can be carried out for any target camera with ease and low cost, offering a practical way of tailoring the training of a DCNN SR model separately to each of the given cameras.

Improving particle detection and size estimation accuracy in digital in-line holography using autoregressive interpolation

The accuracy of particle detection and size estimation is limited by the physical size of the digital sensor used to record the hologram in a digital in-line holographic imaging system. In this paper, we propose to utilize the autoregressive (AR) interpolation of the hologram to increase pixel density and, effectively, the quality of hologram reconstruction. Simulation studies are conducted to evaluate the influence of AR interpolation of a hologram on the accuracy of detection and size estimation of single and multiple particles of varying sizes. A comparative study on the performance of different interpolation techniques indicates the advantage of the proposed AR hologram interpolation approach. An experimental result is provided to validate the suitability of the proposed algorithm in practical applications.

Deep Gradient Prior Regularized Robust Video Super-Resolution

This paper proposes a robust multi-frame video super-resolution (SR) scheme to obtain high SR performance under large upscaling factors. Although the reference low-resolution frames can provide complementary information for the high-resolution frame, an effective regularizer is required to rectify the unreliable information from the reference frames. As the high-frequency information is mostly contained in the image gradient field, we propose to learn the gradient-mapping function between the high-resolution (HR) and the low-resolution (LR) image to regularize the fusion of multiple frames. In contrast to the existing spatial-domain networks, we train a deep gradient-mapping network to learn the horizontal and vertical gradients. We found that adding the low-frequency information (mainly from the LR image) to the gradient-learning network can boost the performance of the network. A forward and backward motion field prior is used to regularize the estimation of the motion flow between frames. For robust SR reconstruction, a weighting scheme is proposed to exclude the outlier data. Visual and quantitative evaluations on benchmark datasets demonstrate that our method is superior to many state-of-the-art methods and can recover better details with less artifacts.

Learning Image Profile Enhancement and Denoising Statistics Priors for Single-Image Super-Resolution

Single-image super-resolution (SR) has been widely used in computer vision applications. The reconstruction-based SR methods are mainly based on certain prior terms to regularize the SR problem. However, it is very challenging to further improve the SR performance by the conventional design of explicit prior terms. Because of the powerful learning ability, deep convolutional neural networks (CNNs) have been widely used in single-image SR task. However, it is difficult to achieve further improvement by only designing the network architecture. In addition, most existing deep CNN-based SR methods learn a nonlinear mapping function to directly map low-resolution (LR) images to desirable high-resolution (HR) images, ignoring the observation models of input images. Inspired by the split Bregman iteration (SBI) algorithm, which is a powerful technique for solving the constrained optimization problems, the original SR problem is divided into two subproblems: 1) inversion subproblem and 2) denoising subproblem. Since the inversion subproblem can be regarded as an inversion step to reconstruct an intermediate HR image with sharper edges and finer structures, we propose to use deep CNN to capture low-level explicit image profile enhancement prior (PEP). Since the denoising subproblem aims to remove the noise in the intermediate image, we adopt a simple and effective denoising network to learn implicit image denoising statistics prior (DSP). Furthermore, the penalty parameter in SBI is adaptively tuned during the iterations for better performance. Finally, we also prove the convergence of our method. Thus, the deep CNNs are exploited to capture both implicit and explicit image statistics priors. Due to SBI, the SR observation model is also leveraged. Consequently, it bridges between two popular SR approaches: 1) learning-based method and 2) reconstruction-based method. Experimental results show that the proposed method achieves the state-of-the-art SR results.

Weighted-encoding-based image interpolation with the nonlocal linear regression model

An image interpolation model based on sparse representation is proposed. Two widely used priors including sparsity and nonlocal self-similarity are used as the regularization terms to boost the performance of the interpolation model. Meanwhile, we incorporate nonlocal linear regression into this model, since nonlocal similar patches could provide a better approximation to a given patch. Moreover, we propose a new approach to learn an adaptive sub-dictionary online instead of clustering. For each patch, similar patches are grouped to learn the adaptive sub-dictionary, generating a more sparse and accurate representation. Finally, weighted encoding is introduced to suppress tailing of fitting residuals in data fidelity. Abundant experimental results show that our proposed method achieves better performance compared to several state-of-the-art methods in terms of subjective and objective evaluations.

Image Interpolation Using Multi-scale Attention-aware Inception Network

A new multi-scale deep learning (MDL) framework is proposed and exploited for conducting image interpolation in this paper. The core of the framework is a seeding network that needs to be designed for the targeted task. For image interpolation, a novel attention-aware inception network (AIN) is developed as the seeding network; it has two key stages: 1) feature extraction based on the low-resolution input image; and 2) feature-to-image mapping to enlarge image's size or resolution. Note that the designed seeding network, AIN, needs to be trained with a matched training dataset at each scale. For that, multi-scale image patches are generated using our proposed pyramid cut, which outperforms the conventional image pyramid method by completely avoiding aliasing issue. After training, the trained AINs are then combined for processing the input image in the testing stage. Extensive experimental simulation results obtained from seven image datasets (comprising 359 images in total) have clearly shown that the proposed MAIN consistently delivers highly accurate interpolated images.

Graph-Based Non-Convex Low-Rank Regularization for Image Compression Artifact Reduction

Block transform coded images usually suffer from annoying artifacts at low bit-rates, because of the independent quantization of DCT coefficients. Image prior models play an important role in compressed image reconstruction. Natural image patches in a small neighborhood of the high-dimensional image space usually exhibit an underlying sub-manifold structure. To model the distribution of signal, we extract sub-manifold structure as prior knowledge. We utilize graph Laplacian regularization to characterize the sub-manifold structure at patch level. And similar patches are exploited as samples to estimate distribution of a particular patch. Instead of using Euclidean distance as similarity metric, we propose to use graph-domain distance to measure the patch similarity. Then we perform low-rank regularization on the similar-patch group, and incorporate a non-convex lp penalty to surrogate matrix rank. Finally, an alternatively minimizing strategy is employed to solve the non-convex problem. Experimental results show that our proposed method is capable of achieving more accurate reconstruction than the state-of-the-art methods in both objective and perceptual qualities.

Depth Restoration From RGB-D Data via Joint Adaptive Regularization and Thresholding on Manifolds

In this paper, we propose a novel depth restoration algorithm from RGB-D data through combining characteristics of local and non-local manifolds, which provide low-dimensional parameterizations of the local and non-local geometry of depth maps. Specifically, on the one hand, a local manifold model is defined to favor local neighboring relationship of pixels in depth, according to which, manifold regularization is introduced to promote smoothing along the manifold structure. On the other hand, the non-local characteristics of the patch-based manifold can be used to build highly data-adaptive orthogonal bases to extract elongated image patterns, accounting for self-similar structures in the manifold. We further define a manifold thresholding operator in 3D adaptive orthogonal spectral bases-eigenvectors of the discrete Laplacian of local and non-local manifolds-to retain only low graph frequencies for depth maps restoration. Finally, we propose a unified alternating direction method of multipliers optimization framework, which elegantly casts the adaptive manifold regularization and thresholding jointly to regularize the inverse problem of depth maps recovery. Experimental results demonstrate that our method achieves superior performance compared with the state-of-the-art works with respect to both objective and subjective quality evaluations.

Effective Content-Aware Chroma Reconstruction Method for Screen Content Images

In this paper, we propose an effective novel content-aware chroma reconstruction (CACR) method for screen content images (SCIs). After receiving the decoded downsampled YUV image on the client side, our fast chroma-copy approach reconstructs the missing chroma pixels in the flat regions of SCI. Then, for non-flat regions, a non-flat region-based winner-first voting (NRWV) strategy is proposed to identify the chroma subsampling scheme used on the server side prior to compression. Further, an effective adaptive hybrid approach is proposed to reconstruct each missing chroma pixel in the non-flat region by fusing the two reconstructed results, one from our modified NRWV-based chroma subsampling-binding and luma-guided chroma reconstruction scheme, which favors the sharp edges in SCI, as well as the other from the bicubic interpolation scheme, which favors blurred and continuous-tone textures. Further, based on the identified chroma subsampling scheme, a geometry alignment-based error compensation approach is proposed to enhance the reconstructed chroma image. Based on typical test SCIs and JCT-VC screen content videos, comprehensive experiments are carried out in HEVC-16.17 to demonstrate that in terms of quality, visual effect, and quality-bitrate tradeoff of the reconstructed SCIs, our CACR method significantly outperforms the existing state-of-the-art methods.

Adaptive Effective Wiener Filter- and Regression-based Upsampling for Asymmetric Resolution Stereoscopic Video Coding

In asymmetric resolution stereoscopic video coding (ARSVC), a stereoscopic video consists of one full-sized leftview video sequence and the synchronized quarter-sized rightview video sequence for achieving a bitrate reduction effect by the encoder. Prior to displaying 3D scenes on the screen, it is necessary to upsample the decoded downsampled right-view video sequence at the client side. In this paper, we propose an effective adaptive upsampling method for ARSVC. First, we employ the resolution- and texture-consistency (RTC) consideration in the conventional Wiener filter-based interpolation scheme, called RTCWF, to enhance the upsampling accuracy in the spatial domain. Second, we propose a linear regression-based interview prediction (LRIP) scheme with residual compensation (RC), called LRIPRC, to increase the upsampling accuracy in the interview domain. Third, we propose an adaptive fusion-based approach to integrate RTCWF and LRIPRC, called RTCWFLRIPRC, to maximize the quality improvement of the upsampled image. Based on seven typical test stereoscopic video sequences, in 3D-HEVC, the experimental results demonstrated that in terms of six well-known quality metrics and execution time requirements, our RTCWF-LRIPRC method outperforms the state-of-the-art upsampling methods for ARSVC.

Fast Image Super-Resolution via Local Adaptive Gradient Field Sharpening Transform

This paper proposes a single-image super-resolution scheme by introducing a gradient field sharpening transform that converts the blurry gradient field of upsampled low-resolution (LR) image to a much sharper gradient field of original high-resolution (HR) image. Different from the existing methods that need to figure out the whole gradient profile structure and locate the edge points, we derive a new approach that sharpens the gradient field adaptively only based on the pixels in a small neighborhood. To maintain image contrast, image gradient is adaptively scaled to keep the integral of gradient field stable. Finally, the HR image is reconstructed by fusing the LR image with the sharpened HR gradient field. Experimental results demonstrate that the proposed algorithm can generate more accurate gradient field and produce super-resolved images with better objective and visual qualities. Another advantage is that the proposed gradient sharpening transform is very fast and suitable for low-complexity applications.

Hallucinating Face Image by Regularization Models in High-Resolution Feature Space

In this paper, we propose two novel regularization models in patch-wise and pixel-wise respectively, which are efficient to reconstruct high-resolution (HR) face image from low-resolution (LR) input. Unlike the conventional patch-based models which depend on the assumption of local geometry consistency in LR and HR spaces, the proposed method directly regularizes the relationship between the target patch and corresponding training set in the HR space. It avoids to deal with the tough problem of preserving local geometry in various resolutions. Taking advantage of kernel function in efficiently describing intrinsic features, we further conduct the patch-based reconstruction model in the high-dimensional kernel space for capturing nonlinear characteristics. Meanwhile, a pixel-based model is proposed to regularize the relationship of pixels in the local neighborhood, which can be employed to enhance the fuzzy details in the target HR face image. It privileges the reconstruction of pixels along the dominant orientation of structure, which is useful for preserving high-frequency information on complex edges. Finally, we combine the two reconstruction models into a unified framework. The output HR face image can be finally optimized by performing an iterative procedure. Experimental results demonstrate that the proposed face hallucination method produces superior performance than the state-of-the-art methods.

Blind Stereoscopic Video Quality Assessment: From Depth Perception to Overall Experience

Stereoscopic video quality assessment (SVQA) is a challenging problem. It has not been well investigated on how to measure depth perception quality independently under different distortion categories and degrees, especially exploit the depth perception to assist the overall quality assessment of 3D videos. In this paper, we propose a new depth perception quality metric (DPQM) and verify that it outperforms existing metrics on our published 3D video extension of High Efficiency Video Coding (3D-HEVC) video database. Furthermore, we validate its effectiveness by applying the crucial part of the DPQM to a novel blind stereoscopic video quality evaluator (BSVQE) for overall 3D video quality assessment. In the DPQM, we introduce the feature of auto-regressive prediction-based disparity entropy (ARDE) measurement and the feature of energy weighted video content measurement, which are inspired by the free-energy principle and the binocular vision mechanism. In the BSVQE, the binocular summation and difference operations are integrated together with the fusion natural scene statistic measurement and the ARDE measurement to reveal the key influence from texture and disparity. Experimental results on three stereoscopic video databases demonstrate that our method outperforms state-of-the-art SVQA algorithms for both symmetrically and asymmetrically distorted stereoscopic video pairs of various distortion types.

Adaptive Chroma Subsampling-Binding and Luma-Guided Chroma Reconstruction Method for Screen Content Images

In this paper, we propose a novel adaptive chroma subsampling-binding and luma-guided (ASBLG) chroma reconstruction method for screen content images (SCIs). After receiving the decoded luma and subsampled chroma image from the decoder, a fast winner-first voting strategy is proposed to identify the used chroma subsampling scheme prior to compression. Then, the decoded luma image is subsampled as the identified subsampling scheme was performed on the chroma image such that we are able to conclude an accurate correlation between the subsampled decoded luma image and the decoded subsampled chroma image. Accordingly, an adaptive sliding window-based and luma-guided chroma reconstruction method is proposed. The related computational complexity analysis is also provided. We take two quality metrics, the color peak signal-to-noise ratio (CPSNR) of the reconstructed chroma images and SCIs and the gradient-based structure similarity index (CGSS) of the reconstructed SCIs to evaluate the quality performance. Let the proposed chroma reconstruction method be denoted as `ASBLG'. Based on 26 typical test SCIs and 6 JCT-VC test screen content video sequences (SCVs), several experiments show that on average, the CPSNR gains of all the reconstructed UV images by 4:2:0(A)-ASBLG, SCIs by 4:2:0(MPEG-B)-ASBLG, and SCVs by 4:2:0(A)-ASBLG are 2.1, 1.87, and 1.87 dB, respectively, when compared with that of the other combinations. Specifically, in terms of CPSNR and CGSS, CSBILINEAR-ASBLG for the test SCIs and CSBICUBIC-ASBLG for the test SCVs outperform the existing state-of-the-art comparative combinations, where CSBILINEAR and CSBICUBIC denote the luma-aware based chroma subsampling schemes by Wang et al.

Dictionary learning based noisy image super-resolution via distance penalty weight model

In this study, we address the problem of noisy image super-resolution. Noisy low resolution (LR) image is always obtained in applications, while most of the existing algorithms assume that the LR image is noise-free. As to this situation, we present an algorithm for noisy image super-resolution which can achieve simultaneously image super-resolution and denoising. And in the training stage of our method, LR example images are noise-free. For different input LR images, even if the noise variance varies, the dictionary pair does not need to be retrained. For the input LR image patch, the corresponding high resolution (HR) image patch is reconstructed through weighted average of similar HR example patches. To reduce computational cost, we use the atoms of learned sparse dictionary as the examples instead of original example patches. We proposed a distance penalty model for calculating the weight, which can complete a second selection on similar atoms at the same time. Moreover, LR example patches removed mean pixel value are also used to learn dictionary rather than just their gradient features. Based on this, we can reconstruct initial estimated HR image and denoised LR image. Combined with iterative back projection, the two reconstructed images are applied to obtain final estimated HR image. We validate our algorithm on natural images and compared with the previously reported algorithms. Experimental results show that our proposed method performs better noise robustness.

Bidirectional texture function image super-resolution using singular value decomposition

The bidirectional texture function (BTF) is widely employed to achieve realistic digital reproduction of real-world material appearance. In practice, a BTF measurement device usually does not use high-resolution (HR) cameras in data collection, considering the high equipment cost and huge data space required. The limited image resolution consequently leads to the loss of texture details in BTF data. This paper proposes a fast BTF image super-resolution (SR) algorithm to deal with this issue. The algorithm uses singular value decomposition (SVD) to separate the collected low-resolution (LR) BTF data into intrinsic textures and eigen-apparent bidirectional reflectance distribution functions (eigen-ABRDFs) and then improves the resolution of the intrinsic textures via image SR. The HR BTFs can be finally obtained by fusing the reconstructed HR intrinsic textures with the LR eigen-ABRDFs. Experimental results show that the proposed algorithm outperforms the state-of-the-art single-image SR algorithms in terms of reconstruction accuracy. In addition, thanks to the employment of SVD, the proposed algorithm is computationally efficient and robust to noise corruption.

Graph-Based Transform for 2D Piecewise Smooth Signals With Random Discontinuity Locations

The graph-based block transform recently emerged as an effective tool for compressing some special signals such as depth images in 3D videos. However, in existing methods, overheads are required to describe the graph of the block, from which the decoder has to calculate the transform via time-consuming eigendecomposition. To address these problems, in this paper, we aim to develop a single graph-based transform for a class of 2D piecewise smooth signals with similar discontinuity patterns. We first consider the deterministic case with a known discontinuity location in each row. We propose a 2D first-order autoregression (2D AR1) model and a 2D graph for this type of signals. We show that the closed-form expression of the inverse of a biased Laplacian matrix of the proposed 2D graph is exactly the covariance matrix of the proposed 2D AR1 model. Therefore, the optimal transform for the signal are the eigenvectors of the proposed graph Laplacian. Next, we show that similar results hold in the random case, where the locations of the discontinuities in different rows are randomly distributed within a confined region, and we derive the closed-form expression of the corresponding optimal 2D graph Laplacian. The theory developed in this paper can be used to design both pre-computed transforms and signal-dependent transforms with low complexities. Finally, depth image coding experiments demonstrate that our methods can achieve similar performance to the state-of-the-art method, but our complexity is much lower.

Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain

Inverse imaging problems are inherently underdetermined, and hence, it is important to employ appropriate image priors for regularization. One recent popular prior-the graph Laplacian regularizer-assumes that the target pixel patch is smooth with respect to an appropriately chosen graph. However, the mechanisms and implications of imposing the graph Laplacian regularizer on the original inverse problem are not well understood. To address this problem, in this paper, we interpret neighborhood graphs of pixel patches as discrete counterparts of Riemannian manifolds and perform analysis in the continuous domain, providing insights into several fundamental aspects of graph Laplacian regularization for image denoising. Specifically, we first show the convergence of the graph Laplacian regularizer to a continuous-domain functional, integrating a norm measured in a locally adaptive metric space. Focusing on image denoising, we derive an optimal metric space assuming non-local self-similarity of pixel patches, leading to an optimal graph Laplacian regularizer for denoising in the discrete domain. We then interpret graph Laplacian regularization as an anisotropic diffusion scheme to explain its behavior during iterations, e.g., its tendency to promote piecewise smooth signals under certain settings. To verify our analysis, an iterative image denoising algorithm is developed. Experimental results show that our algorithm performs competitively with state-of-the-art denoising methods, such as BM3D for natural images, and outperforms them significantly for piecewise smooth images.

Fast Single Image Super-Resolution Using a New Analytical Solution for l2 - l2 Problems

This paper addresses the problem of single image super-resolution (SR), which consists of recovering a high-resolution image from its blurred, decimated, and noisy version. The existing algorithms for single image SR use different strategies to handle the decimation and blurring operators. In addition to the traditional first-order gradient methods, recent techniques investigate splitting-based methods dividing the SR problem into up-sampling and deconvolution steps that can be easily solved. Instead of following this splitting strategy, we propose to deal with the decimation and blurring operators simultaneously by taking advantage of their particular properties in the frequency domain, leading to a new fast SR approach. Specifically, an analytical solution is derived and implemented efficiently for the Gaussian prior or any other regularization that can be formulated into an l2 -regularized quadratic model, i.e., an l2 - l2 optimization problem. The flexibility of the proposed SR scheme is shown through the use of various priors/regularizations, ranging from generic image priors to learning-based approaches. In the case of non-Gaussian priors, we show how the analytical solution derived from the Gaussian case can be embedded into traditional splitting frameworks, allowing the computation cost of existing algorithms to be decreased significantly. Simulation results conducted on several images with different priors illustrate the effectiveness of our fast SR approach compared with existing techniques.

Single Image Super-Resolution Using Local Geometric Duality and Non-Local Similarity

Super-resolution (SR) from a single image plays an important role in many computer vision applications. It aims to estimate a high-resolution (HR) image from an input low- resolution (LR) image. To ensure a reliable and robust estimation of the HR image, we propose a novel single image SR method that exploits both the local geometric duality (GD) and the non-local similarity of images. The main principle is to formulate these two typically existing features of images as effective priors to constrain the super-resolved results. In consideration of this principle, the robust soft-decision interpolation method is generalized as an outstanding adaptive GD (AGD)-based local prior. To adaptively design weights for the AGD prior, a local non-smoothness detection method and a directional standard-deviation-based weights selection method are proposed. After that, the AGD prior is combined with a variational-framework-based non-local prior. Furthermore, the proposed algorithm is speeded up by a fast GD matrices construction method, which primarily relies on the selective pixel processing. The extensive experimental results verify the effectiveness of the proposed method compared with several state-of-the-art SR algorithms.

Video Compression Artifact Reduction via Spatio-Temporal Multi-Hypothesis Prediction

Annoying compression artifacts exist in most of lossy coded videos at low bit rates, which are caused by coarse quantization of transform coefficients or motion compensation from distorted frames. In this paper, we propose a compression artifact reduction approach that utilizes both the spatial and the temporal correlation to form multi-hypothesis predictions from spatio-temporal similar blocks. For each transform block, three predictions with their reliabilities are estimated, respectively. The first prediction is constructed by inversely quantizing transform coefficients directly, and its reliability is determined by the variance of quantization noise. The second prediction is derived by representing each transform block with a temporal auto-regressive (TAR) model along its motion trajectory, and its corresponding reliability is estimated from local prediction errors of the TAR model. The last prediction infers the original coefficients from similar blocks in non-local regions, and its reliability is estimated based on the distribution of coefficients in these similar blocks. Finally, all the predictions are adaptively fused according to their reliabilities to restore high-quality videos. The experimental results show that the proposed method can efficiently reduce most of the compression artifacts and improve both subjective and objective quality of block transform coded videos.

Objective Quality Assessment of Interpolated Natural Images

Image interpolation techniques that create high-resolution images from low-resolution (LR) images are widely used in real world applications, but how to evaluate the quality of interpolated images is not a well-resolved issue. Subjective assessment methods are useful and reliable, but are also slow and expensive. Here, we propose an objective method to assess the quality of an interpolated natural image using the available LR image as a reference. Our method adopts a natural scene statistics (NSS) framework, where image quality degradation is gauged by the deviation of its statistical features from the NSS models trained upon high-quality natural images. Two distortion measures are proposed, namely, interpolated natural image distortion (IND) and weighted IND. Validations by subjective tests show that the proposed approach performs statistically equivalent or sometimes better than an average human subject. Moreover, we demonstrate the potential application of the proposed method in parameter tuning of image interpolation algorithms.

Fast image interpolation via random forests

This paper proposes a two-stage framework for fast image interpolation via random forests (FIRF). The proposed FIRF method gives high accuracy, as well as requires low computation. The underlying idea of this proposed work is to apply random forests to classify the natural image patch space into numerous subspaces and learn a linear regression model for each subspace to map the low-resolution image patch to high-resolution image patch. The FIRF framework consists of two stages. Stage 1 of the framework removes most of the ringing and aliasing artifacts in the initial bicubic interpolated image, while Stage 2 further refines the Stage 1 interpolated image. By varying the number of decision trees in the random forests and the number of stages applied, the proposed FIRF method can realize computationally scalable image interpolation. Extensive experimental results show that the proposed FIRF(3, 2) method achieves more than 0.3 dB improvement in peak signal-to-noise ratio over the state-of-the-art nonlocal autoregressive modeling (NARM) method. Moreover, the proposed FIRF(1, 1) obtains similar or better results as NARM while only takes its 0.3% computational time.

Single image superresolution based on gradient profile sharpness

Single image superresolution is a classic and active image processing problem, which aims to generate a high-resolution (HR) image from a low-resolution input image. Due to the severely under-determined nature of this problem, an effective image prior is necessary to make the problem solvable, and to improve the quality of generated images. In this paper, a novel image superresolution algorithm is proposed based on gradient profile sharpness (GPS). GPS is an edge sharpness metric, which is extracted from two gradient description models, i.e., a triangle model and a Gaussian mixture model for the description of different kinds of gradient profiles. Then, the transformation relationship of GPSs in different image resolutions is studied statistically, and the parameter of the relationship is estimated automatically. Based on the estimated GPS transformation relationship, two gradient profile transformation models are proposed for two profile description models, which can keep profile shape and profile gradient magnitude sum consistent during profile transformation. Finally, the target gradient field of HR image is generated from the transformed gradient profiles, which is added as the image prior in HR image reconstruction model. Extensive experiments are conducted to evaluate the proposed algorithm in subjective visual effect, objective quality, and computation time. The experimental results demonstrate that the proposed approach can generate superior HR images with better visual quality, lower reconstruction error, and acceptable computation efficiency as compared with state-of-the-art works.

Single Image Superresolution via Directional Group Sparsity and Directional Features

Single image superresolution (SR) aims to construct a high-resolution version from a single low-resolution (LR) image. The SR reconstruction is challenging because of the missing details in the given LR image. Thus, it is critical to explore and exploit effective prior knowledge for boosting the reconstruction performance. In this paper, we propose a novel SR method by exploiting both the directional group sparsity of the image gradients and the directional features in similarity weight estimation. The proposed SR approach is based on two observations: 1) most of the sharp edges are oriented in a limited number of directions and 2) an image pixel can be estimated by the weighted averaging of its neighbors. In consideration of these observations, we apply the curvelet transform to extract directional features which are then used for region selection and weight estimation. A combined total variation regularizer is presented which assumes that the gradients in natural images have a straightforward group sparsity structure. In addition, a directional nonlocal means regularization term takes pixel values and directional information into account to suppress unwanted artifacts. By assembling the designed regularization terms, we solve the SR problem of an energy function with minimal reconstruction error by applying a framework of templates for first-order conic solvers. The thorough quantitative and qualitative results in terms of peak signal-to-noise ratio, structural similarity, information fidelity criterion, and preference matrix demonstrate that the proposed approach achieves higher quality SR reconstruction than the state-of-the-art algorithms.

Using compressive sensing to recover images from PET scanners with partial detector rings

PURPOSE

Most positron emission tomography/computed tomography (PET/CT) scanners consist of tightly packed discrete detector rings to improve scanner efficiency. The authors' aim was to use compressive sensing (CS) techniques in PET imaging to investigate the possibility of decreasing the number of detector elements per ring (introducing gaps) while maintaining image quality.

METHODS

A CS model based on a combination of gradient magnitude and wavelet domains (wavelet-TV) was developed to recover missing observations in PET data acquisition. The model was designed to minimize the total variation (TV) and L1-norm of wavelet coefficients while constrained by the partially observed data. The CS model also incorporated a Poisson noise term that modeled the observed noise while suppressing its contribution by penalizing the Poisson log likelihood function. Three experiments were performed to evaluate the proposed CS recovery algorithm: a simulation study, a phantom study, and six patient studies. The simulation dataset comprised six disks of various sizes in a uniform background with an activity concentration of 5:1. The simulated image was multiplied by the system matrix to obtain the corresponding sinogram and then Poisson noise was added. The resultant sinogram was masked to create the effect of partial detector removal and then the proposed CS algorithm was applied to recover the missing PET data. In addition, different levels of noise were simulated to assess the performance of the proposed algorithm. For the phantom study, an IEC phantom with six internal spheres each filled with F-18 at an activity-to-background ratio of 10:1 was used. The phantom was imaged twice on a RX PET/CT scanner: once with all detectors operational (baseline) and once with four detector blocks (11%) turned off at each of 0 ˚, 90 ˚, 180 ˚, and 270° (partially sampled). The partially acquired sinograms were then recovered using the proposed algorithm. For the third test, PET images from six patient studies were investigated using the same strategy of the phantom study. The recovered images using WTV and TV as well as the partially sampled images from all three experiments were then compared with the fully sampled images (the baseline). Comparisons were done by calculating the mean error (%bias), root mean square error (RMSE), contrast recovery (CR), and SNR of activity concentration in regions of interest drawn in the background as well as the disks, spheres, and lesions.

RESULTS

For the simulation study, the mean error, RMSE, and CR for the WTV (TV) recovered images were 0.26% (0.48%), 2.6% (2.9%), 97% (96%), respectively, when compared to baseline. For the partially sampled images, these results were 22.5%, 45.9%, and 64%, respectively. For the simulation study, the average SNR for the baseline was 41.7 while for WTV (TV), recovered image was 44.2 (44.0). The phantom study showed similar trends with 5.4% (18.2%), 15.6% (18.8%), and 78% (60%), respectively, for the WTV (TV) images and 33%, 34.3%, and 69% for the partially sampled images. For the phantom study, the average SNR for the baseline was 14.7 while for WTV (TV) recovered image was 13.7 (11.9). Finally, the average of these values for the six patient studies for the WTV-recovered, TV, and partially sampled images was 1%, 7.2%, 92% and 1.3%, 15.1%, 87%, and 27%, 25.8%, 45%, respectively.

CONCLUSIONS

CS with WTV is capable of recovering PET images with good quantitative accuracy from partially sampled data. Such an approach can be used to potentially reduce the cost of scanners while maintaining good image quality.

Dictionary learning for data recovery in positron emission tomography

Compressed sensing (CS) aims to recover images from fewer measurements than that governed by the Nyquist sampling theorem. Most CS methods use analytical predefined sparsifying domains such as total variation, wavelets, curvelets, and finite transforms to perform this task. In this study, we evaluated the use of dictionary learning (DL) as a sparsifying domain to reconstruct PET images from partially sampled data, and compared the results to the partially and fully sampled image (baseline).A CS model based on learning an adaptive dictionary over image patches was developed to recover missing observations in PET data acquisition. The recovery was done iteratively in two steps: a dictionary learning step and an image reconstruction step. Two experiments were performed to evaluate the proposed CS recovery algorithm: an IEC phantom study and five patient studies. In each case, 11% of the detectors of a GE PET/CT system were removed and the acquired sinogram data were recovered using the proposed DL algorithm. The recovered images (DL) as well as the partially sampled images (with detector gaps) for both experiments were then compared to the baseline. Comparisons were done by calculating RMSE, contrast recovery and SNR in ROIs drawn in the background, and spheres of the phantom as well as patient lesions.For the phantom experiment, the RMSE for the DL recovered images were 5.8% when compared with the baseline images while it was 17.5% for the partially sampled images. In the patients' studies, RMSE for the DL recovered images were 3.8%, while it was 11.3% for the partially sampled images. Our proposed CS with DL is a good approach to recover partially sampled PET data. This approach has implications toward reducing scanner cost while maintaining accurate PET image quantification.

Wavelet Domain Multidictionary Learning for Single Image Super-Resolution

Image super-resolution (SR) aims at recovering the high-frequency (HF) details of a high-resolution (HR) image according to the given low-resolution (LR) image and some priors about natural images. Learning the relationship of the LR image and its corresponding HF details to guide the reconstruction of the HR image is needed. In order to alleviate the uncertainty in HF detail prediction, the HR and LR images are usually decomposed into 4 subbands after 1-level discrete wavelet transformation (DWT), including an approximation subband and three detail subbands. From our observation, we found the approximation subbands of the HR image and the corresponding bicubic interpolated image are very similar but the respective detail subbands are different. Therefore, an algorithm to learn 4 coupled principal component analysis (PCA) dictionaries to describe the relationship between the approximation subband and the detail subbands is proposed in this paper. Comparisons with various state-of-the-art methods by experiments showed that our proposed algorithm is superior to some related works.

Fast image upsampling via the displacement field

In this paper, we present a fast image upsampling method within a two-scale framework to ensure the sharp construction of upsampled image for both large-scale edges and small-scale structures. In our approach, the low-frequency image is recovered via a novel sharpness preserving interpolation technique based on a well-constructed displacement field, which is estimated by a cross-resolution sharpness preserving model. Within this model, the distances of pixels on edges are preserved, which enables the recovery of sharp edges in the high-resolution result. Likewise, local high-frequency structures are reconstructed via a sharpness preserving reconstruction algorithm. Extensive experiments show that our method outperforms current state-of-the-art approaches, based on quantitative and qualitative evaluations, as well as perceptual evaluation by a user study. Moreover, our approach is very fast so as to be practical for real applications.

Color-guided depth recovery from RGB-D data using an adaptive autoregressive model

This paper proposes an adaptive color-guided autoregressive (AR) model for high quality depth recovery from low quality measurements captured by depth cameras. We observe and verify that the AR model tightly fits depth maps of generic scenes. The depth recovery task is formulated into a minimization of AR prediction errors subject to measurement consistency. The AR predictor for each pixel is constructed according to both the local correlation in the initial depth map and the nonlocal similarity in the accompanied high quality color image. We analyze the stability of our method from a linear system point of view, and design a parameter adaptation scheme to achieve stable and accurate depth recovery. Quantitative and qualitative evaluation compared with ten state-of-the-art schemes show the effectiveness and superiority of our method. Being able to handle various types of depth degradations, the proposed method is versatile for mainstream depth sensors, time-of-flight camera, and Kinect, as demonstrated by experiments on real systems.

Single image interpolation via adaptive nonlocal sparsity-based modeling

Single image interpolation is a central and extensively studied problem in image processing. A common approach toward the treatment of this problem in recent years is to divide the given image into overlapping patches and process each of them based on a model for natural image patches. Adaptive sparse representation modeling is one such promising image prior, which has been shown to be powerful in filling-in missing pixels in an image. Another force that such algorithms may use is the self-similarity that exists within natural images. Processing groups of related patches together exploits their correspondence, leading often times to improved results. In this paper, we propose a novel image interpolation method, which combines these two forces-nonlocal self-similarities and sparse representation modeling. The proposed method is contrasted with competitive and related algorithms, and demonstrated to achieve state-of-the-art results.