Blind and semi-blind deblurring of natural images

Research on Blind Super-Resolution Technology for Infrared Images of Power Equipment Based on Compressed Sensing Theory

Infrared images of power equipment play an important role in power equipment status monitoring and fault identification. Aiming to resolve the problems of low resolution and insufficient clarity in the application of infrared images, we propose a blind super-resolution algorithm based on the theory of compressed sensing. It includes an improved blur kernel estimation method combined with compressed sensing theory and an improved infrared image super-resolution reconstruction algorithm based on block compressed sensing theory. In the blur kernel estimation method, we propose a blur kernel estimation algorithm under the compressed sensing framework to realize the estimation of the blur kernel from low-resolution images. In the estimation process, we define a new Lw norm to constrain the gradient image in the iterative process by analyzing the significant edge intensity changes before and after the image is blurred. With the Lw norm, the salient edges can be selected and enhanced, the intermediate latent image generated by the iteration can move closer to the clear image, and the accuracy of the blur kernel estimation can be improved. For the super-resolution reconstruction algorithm, we introduce a blur matrix and a regular total variation term into the traditional compressed sensing model and design a two-step total variation sparse iteration (TwTVSI) algorithm. Therefore, while ensuring the computational efficiency, the boundary effect caused by the block processing inside the image is removed. In addition, the design of the TwTVSI algorithm can effectively process the super-resolution model of compressed sensing with a sparse dictionary, thereby breaking through the reconstruction performance limitation of the traditional regularized super-resolution method of compressed sensing due to the lack of sparseness in the signal transform domain. The final experimental results also verify the effectiveness of our blind super-resolution algorithm.

Effective Blind Image Deblurring Using Matrix-Variable Optimization

Blind image deblurring has been a challenging issue due to the unknown blur and computation problem. Recently, the matrix-variable optimization method successfully demonstrates its potential advantages in computation. This paper proposes an effective matrix-variable optimization method for blind image deblurring. Blur kernel matrix is exactly decomposed by a direct SVD technique. The blur kernel and original image are well estimated by minimizing a matrix-variable optimization problem with blur kernel constraints. A matrix-type alternative iterative algorithm is proposed to solve the matrix-variable optimization problem. Finally, experimental results show that the proposed blind image deblurring method is much superior to the state-of-the-art blind image deblurring algorithms in terms of image quality and computation time.

Learning Wavefront Coding for Extended Depth of Field Imaging

Depth of field is an important factor of imaging systems that highly affects the quality of the acquired spatial information. Extended depth of field (EDoF) imaging is a challenging ill-posed problem and has been extensively addressed in the literature. We propose a computational imaging approach for EDoF, where we employ wavefront coding via a diffractive optical element (DOE) and we achieve deblurring through a convolutional neural network. Thanks to the end-to-end differentiable modeling of optical image formation and computational post-processing, we jointly optimize the optical design, i.e., DOE, and the deblurring through standard gradient descent methods. Based on the properties of the underlying refractive lens and the desired EDoF range, we provide an analytical expression for the search space of the DOE, which is instrumental in the convergence of the end-to-end network. We achieve superior EDoF imaging performance compared to the state of the art, where we demonstrate results with minimal artifacts in various scenarios, including deep 3D scenes and broadband imaging.

Blind Deconvolution Based on Compressed Sensing with bi-l0-l2-norm Regularization in Light Microscopy Image

Blind deconvolution of light microscopy images could improve the ability of distinguishing cell-level substances. In this study, we investigated the blind deconvolution framework for a light microscope image, which combines the benefits of bi-l₀-l₂-norm regularization with compressed sensing and conjugated gradient algorithms. Several existing regularization approaches were limited by staircase artifacts (or cartooned artifacts) and noise amplification. Thus, we implemented our strategy to overcome these problems using the bi-l₀-l₂-norm regularization proposed. It was investigated through simulations and experiments using optical microscopy images including the background noise. The sharpness was improved through the successful image restoration while minimizing the noise amplification. In addition, quantitative factors of the restored images, including the intensity profile, root-mean-square error (RMSE), edge preservation index (EPI), structural similarity index measure (SSIM), and normalized noise power spectrum, were improved compared to those of existing or comparative images. In particular, the results of using the proposed method showed RMSE, EPI, and SSIM values of approximately 0.12, 0.81, and 0.88 when compared with the reference. In addition, RMSE, EPI, and SSIM values in the restored image were proven to be improved by about 5.97, 1.26, and 1.61 times compared with the degraded image. Consequently, the proposed method is expected to be effective for image restoration and to reduce the cost of a high-performance light microscope.

Graph-Based Blind Image Deblurring From a Single Photograph

Blind image deblurring, i.e., deblurring without knowledge of the blur kernel, is a highly ill-posed problem. The problem can be solved in two parts: i) estimate a blur kernel from the blurry image, and ii) given an estimated blur kernel, de-convolve the blurry input to restore the target image. In this paper, we propose a graph-based blind image deblurring algorithm by interpreting an image patch as a signal on a weighted graph. Specifically, we first argue that a skeleton image-a proxy that retains the strong gradients of the target but smooths out the details-can be used to accurately estimate the blur kernel and has a unique bi-modal edge weight distribution. Then, we design a reweighted graph total variation (RGTV) prior that can efficiently promote a bi-modal edge weight distribution given a blurry patch. Further, to analyze RGTV in the graph frequency domain, we introduce a new weight function to represent RGTV as a graph l1-Laplacian regularizer. This leads to a graph spectral filtering interpretation of the prior with desirable properties, including robustness to noise and blur, strong piecewise smooth (PWS) filtering and sharpness promotion. Minimizing a blind image deblurring objective with RGTV results in a non-convex non-differentiable optimization problem. Leveraging the new graph spectral interpretation for RGTV, we design an efficient algorithm that solves for the skeleton image and the blur kernel alternately. Specifically for Gaussian blur, we propose a further speedup strategy for blind Gaussian deblurring using accelerated graph spectral filtering. Finally, with the computed blur kernel, recent non-blind image deblurring algorithms can be applied to restore the target image. Experimental results demonstrate that our algorithm successfully restores latent sharp images and outperforms state-of-the-art methods quantitatively and qualitatively.

Blind Restoration Algorithm Using Residual Measures for Motion-Blurred Noisy Images

Image de-blurring is an inverse problem whose intent is to recover an image from the image affected badly with different environmental conditions. Usually, blurring can happen in various ways; however, de-blurring from a motion problem with or without noise can pose an important problem that is difficult to solve with less computation task. The quality of the restored image in iterative methods of blind motion de-blurring depends on the regularization parameter and the iteration number, which can be automatically or manually stopped. Blind de-blurring and restoration employing image de-blurring and whiteness measures are proposed in this paper to automatically decide the number of iterations. The technique has three modules, namely image de-blurring module, whiteness measures module, and image estimation module. New whiteness measures of hole entropy and mean-square contingency coefficient have been proposed in the whiteness measures module. Initially, the blurred image is de-blurred by the employment of edge responses and image priors using point-spread function. Later, whiteness measures are computed for the de-blurred image and, finally, the best image is selected. The results are obtained for all eight whiteness measures by employing evaluation metrics of increase in signal-to-noise ratio (ISNR), mean-square error, and structural similarity index. The results are obtained from standard images, and performance analysis is made by varying parameters. The obtained results for synthetically blurred images are good even under a noisy condition with ΔISNR average values of 0.3066 dB. The proposed whiteness measures seek a powerful solution to iterative de-blurring algorithms in deciding automatic stopping criteria.

Combining Motion Compensation with Spatiotemporal Constraint for Video Deblurring

We propose a video deblurring method by combining motion compensation with spatiotemporal constraint for restoring blurry video caused by camera shake. The proposed method makes effective full use of the spatiotemporal information not only in the blur kernel estimation, but also in the latent sharp frame restoration. Firstly, we estimate a motion vector between the current and the previous blurred frames, and introduce the estimated motion vector for deriving the motion-compensated frame with the previous restored frame. Secondly, we proposed a blur kernel estimation strategy by applying the derived motion-compensated frame to an improved regularization model for improving the quality of the estimated blur kernel and reducing the processing time. Thirdly, we propose a spatiotemporal constraint algorithm that can not only enhance temporal consistency, but also suppress noise and ringing artifacts of the deblurred video through introducing a temporal regularization term. Finally, we extend Fast Total Variation de-convolution (FTVd) for solving the minimization problem of the proposed spatiotemporal constraint energy function. Extensive experiments demonstrate that the proposed method achieve the state-of-the-art results either in subjective vision or objective evaluation.

Explicit Ringing Removal in Image Deblurring

In this paper, we present a simple yet effective image deblurring method to produce ringing-free deblurred images. Our work is inspired by the observation that large-scale deblurring ringing artifacts are measurable through a multi-resolution pyramid of low-pass filtering of the blurred-deblurred image pair. We propose to model such a quantification as a convex cost function and minimize it directly in the deblurring process in order to reduce ringing regardless of its cause. An efficient primal-dual algorithm is proposed as a solution to this optimization problem. Since the regularization is more biased toward ringing patterns, the details of the reconstructed image are prevented from over-smoothing. An inevitable source of ringing is sensor saturation which can be detected costlessly contrary to most other sources of ringing. However, dealing with the saturation effect in deblurring introduces a non-linear operator in optimization problem. In this paper, we also introduce a linear approximation as a solution to handling saturation in the proposed deblurring method. As a result of these steps, we significantly enhance the quality of the deblurred images. Experimental results and quantitative evaluations demonstrate that the proposed method performs favorably against state-of-the-art image deblurring methods.

Blind Deconvolution With Model Discrepancies

Blind deconvolution is a strongly ill-posed problem comprising of simultaneous blur and image estimation. Recent advances in prior modeling and/or inference methodology led to methods that started to perform reasonably well in real cases. However, as we show here, they tend to fail if the convolution model is violated even in a small part of the image. Methods based on variational Bayesian inference play a prominent role. In this paper, we use this inference in combination with the same prior for noise, image, and blur that belongs to the family of independent non-identical Gaussian distributions, known as the automatic relevance determination prior. We identify several important properties of this prior useful in blind deconvolution, namely, enforcing non-negativity of the blur kernel, favoring sharp images over blurred ones, and most importantly, handling non-Gaussian noise, which, as we demonstrate, is common in real scenarios. The presented method handles discrepancies in the convolution model, and thus extends applicability of blind deconvolution to real scenarios, such as photos blurred by camera motion and incorrect focus.

Blind Image Blur Estimation via Deep Learning

Image blur kernel estimation is critical to blind image deblurring. Most existing approaches exploit handcrafted blur features that are optimized for a certain uniform blur across the image, which is unrealistic in a real blind deconvolution setting, where the blur type is often unknown. To deal with this issue, we aim at identifying the blur type for each input image patch, and then estimating the kernel parameter in this paper. A learning-based method using a pre-trained deep neural network (DNN) and a general regression neural network (GRNN) is proposed to first classify the blur type and then estimate its parameters, taking advantages of both the classification ability of DNN and the regression ability of GRNN. To the best of our knowledge, this is the first time that pre-trained DNN and GRNN have been applied to the problem of blur analysis. First, our method identifies the blur type from a mixed input of image patches corrupted by various blurs with different parameters. To this aim, a supervised DNN is trained to project the input samples into a discriminative feature space, in which the blur type can be easily classified. Then, for each blur type, the proposed GRNN estimates the blur parameters with very high accuracy. Experiments demonstrate the effectiveness of the proposed method in several tasks with better or competitive results compared with the state of the art on two standard image data sets, i.e., the Berkeley segmentation data set and the Pascal VOC 2007 data set. In addition, blur region segmentation and deblurring on a number of real photographs show that our method outperforms the previous techniques even for non-uniformly blurred images.

Closed-Loop Restoration Approach to Blurry Images Based on Machine Learning and Feedback Optimization

Blind image deconvolution (BID) aims to remove or reduce the degradations that have occurred during the acquisition or processing. It is a challenging ill-posed problem due to a lack of enough information in degraded image for unambiguous recovery of both point spread function (PSF) and clear image. Although recently many powerful algorithms appeared; however, it is still an active research area due to the diversity of degraded images as well as degradations. Closed-loop control systems are characterized with their powerful ability to stabilize the behavior response and overcome external disturbances by designing an effective feedback optimization. In this paper, we employed feedback control to enhance the stability of BID by driving the current estimation quality of PSF to the desired level without manually selecting restoration parameters and using an effective combination of machine learning with feedback optimization. The foremost challenge when designing a feedback structure is to construct or choose a suitable performance metric as a controlled index and a feedback information. Our proposed quality metric is based on the blur assessment of deconvolved patches to identify the best PSF and computing its relative quality. The Kalman filter-based extremum seeking approach is employed to find the optimum value of controlled variable. To find better restoration parameters, learning algorithms, such as multilayer perceptron and bagged decision trees, are used to estimate the generic PSF support size instead of trial and error methods. The problem is modeled as a combination of pattern classification and regression using multiple training features, including noise metrics, blur metrics, and low-level statistics. Multi-objective genetic algorithm is used to find key patches from multiple saliency maps which enhance performance and save extra computation by avoiding ineffectual regions of the image. The proposed scheme is shown to outperform corresponding open-loop schemes, which often fails or needs many assumptions regarding images and thus resulting in sub-optimal results.

Deblurring Saturated Night Image With Function-Form Kernel

Deblurring saturated night images are a challenging problem because such images have low contrast combined with heavy noise and saturated regions. Unlike the deblurring schemes that discard saturated regions when estimating blur kernels, this paper proposes a novel scheme to deduce blur kernels from saturated regions via a novel kernel representation and advanced algorithms. Our key technical contribution is the proposed function-form representation of blur kernels, which regularizes existing matrix-form kernels using three functional components: 1) trajectory; 2) intensity; and 3) expansion. From automatically detected saturated regions, their skeleton, brightness, and width are fitted into the corresponding three functional components of blur kernels. Such regularization significantly improves the quality of kernels deduced from saturated regions. Second, we propose an energy minimizing algorithm to select and assign the deduced function-form kernels to partitioned image regions as the initialization for non-uniform deblurring. Finally, we convert the assigned function-form kernels into matrix form for more detailed estimation in a multi-scale deconvolution. Experimental results show that our scheme outperforms existing schemes on challenging real examples.

Computational multi-focus imaging combining sparse model with color dependent phase mask

A method for extended depth of field imaging based on image acquisition through a thin binary phase plate followed by fast automatic computational post-processing is presented. By placing a wavelength dependent optical mask inside the pupil of a conventional camera lens, one acquires a unique response for each of the three main color channels, which adds valuable information that allows blind reconstruction of blurred images without the need of an iterative search process for estimating the blurring kernel. The presented simulation as well as capture of a real life scene show how acquiring a one-shot image focused at a single plane, enable generating a de-blurred scene over an extended range in space.

A New Study of Blind Deconvolution with Implicit Incorporation of Nonnegativity Constraints

The inverse problem of image restoration to remove noise and blur in an observed image was extensively studied in the last two decades. For the case of a known blurring kernel (or a known blurring type such as out of focus or Gaussian blur), many effective models and efficient solvers exist. However when the underlying blur is unknown, there have been fewer developments for modelling the so-called blind deblurring since the early works of You and Kaveh (1996) and Chan and Wong (1998). A major challenge is how to impose the extra constraints to ensure quality of restoration. This paper proposes a new transform based method to impose the positivity constraints automatically and then two numerical solution algorithms. Test results demonstrate the effectiveness and robustness of the proposed method in restoring blurred images.

A novel SURE-based criterion for parametric PSF estimation

We propose an unbiased estimate of a filtered version of the mean squared error--the blur-SURE (Stein's unbiased risk estimate)--as a novel criterion for estimating an unknown point spread function (PSF) from the degraded image only. The PSF is obtained by minimizing this new objective functional over a family of Wiener processings. Based on this estimated blur kernel, we then perform nonblind deconvolution using our recently developed algorithm. The SURE-based framework is exemplified with a number of parametric PSF, involving a scaling factor that controls the blur size. A typical example of such parametrization is the Gaussian kernel. The experimental results demonstrate that minimizing the blur-SURE yields highly accurate estimates of the PSF parameters, which also result in a restoration quality that is very similar to the one obtained with the exact PSF, when plugged into our recent multi-Wiener SURE-LET deconvolution algorithm. The highly competitive results obtained outline the great potential of developing more powerful blind deconvolution algorithms based on SURE-like estimates.

Parametric blur estimation for blind restoration of natural images: linear motion and out-of-focus

This paper presents a new method to estimate the parameters of two types of blurs, linear uniform motion (approximated by a line characterized by angle and length) and out-of-focus (modeled as a uniform disk characterized by its radius), for blind restoration of natural images. The method is based on the spectrum of the blurred images and is supported on a weak assumption, which is valid for the most natural images: the power-spectrum is approximately isotropic and has a power-law decay with the spatial frequency. We introduce two modifications to the radon transform, which allow the identification of the blur spectrum pattern of the two types of blurs above mentioned. The blur parameters are identified by fitting an appropriate function that accounts separately for the natural image spectrum and the blur frequency response. The accuracy of the proposed method is validated by simulations, and the effectiveness of the proposed method is assessed by testing the algorithm on real natural blurred images and comparing it with state-of-the-art blind deconvolution methods.

Parameter estimation for blind and non-blind deblurring using residual whiteness measures

Image deblurring (ID) is an ill-posed problem typically addressed by using regularization, or prior knowledge, on the unknown image (and also on the blur operator, in the blind case). ID is often formulated as an optimization problem, where the objective function includes a data term encouraging the estimated image (and blur, in blind ID) to explain the observed data well (typically, the squared norm of a residual) plus a regularizer that penalizes solutions deemed undesirable. The performance of this approach depends critically (among other things) on the relative weight of the regularizer (the regularization parameter) and on the number of iterations of the algorithm used to address the optimization problem. In this paper, we propose new criteria for adjusting the regularization parameter and/or the number of iterations of ID algorithms. The rationale is that if the recovered image (and blur, in blind ID) is well estimated, the residual image is spectrally white; contrarily, a poorly deblurred image typically exhibits structured artifacts (e.g., ringing, oversmoothness), yielding residuals that are not spectrally white. The proposed criterion is particularly well suited to a recent blind ID algorithm that uses continuation, i.e., slowly decreases the regularization parameter along the iterations; in this case, choosing this parameter and deciding when to stop are one and the same thing. Our experiments show that the proposed whiteness-based criteria yield improvements in SNR, on average, only 0.15 dB below those obtained by (clairvoyantly) stopping the algorithm at the best SNR. We also illustrate the proposed criteria on non-blind ID, reporting results that are competitive with state-of-the-art criteria (such as Monte Carlo-based GSURE and projected SURE), which, however, are not applicable for blind ID.

Unified blind method for multi-image super-resolution and single/multi-image blur deconvolution

This paper presents, for the first time, a unified blind method for multi-image super-resolution (MISR or SR), single-image blur deconvolution (SIBD), and multi-image blur deconvolution (MIBD) of low-resolution (LR) images degraded by linear space-invariant (LSI) blur, aliasing, and additive white Gaussian noise (AWGN). The proposed approach is based on alternating minimization (AM) of a new cost function with respect to the unknown high-resolution (HR) image and blurs. The regularization term for the HR image is based upon the Huber-Markov random field (HMRF) model, which is a type of variational integral that exploits the piecewise smooth nature of the HR image. The blur estimation process is supported by an edge-emphasizing smoothing operation, which improves the quality of blur estimates by enhancing strong soft edges toward step edges, while filtering out weak structures. The parameters are updated gradually so that the number of salient edges used for blur estimation increases at each iteration. For better performance, the blur estimation is done in the filter domain rather than the pixel domain, i.e., using the gradients of the LR and HR images. The regularization term for the blur is Gaussian (L2 norm), which allows for fast noniterative optimization in the frequency domain. We accelerate the processing time of SR reconstruction by separating the upsampling and registration processes from the optimization procedure. Simulation results on both synthetic and real-life images (from a novel computational imager) confirm the robustness and effectiveness of the proposed method.

Video deblurring algorithm using accurate blur kernel estimation and residual deconvolution based on a blurred-unblurred frame pair

Blurred frames may happen sparsely in a video sequence acquired by consumer devices such as digital camcorders and digital cameras. In order to avoid visually annoying artifacts due to those blurred frames, this paper presents a novel motion deblurring algorithm in which a blurred frame can be reconstructed utilizing the high-resolution information of adjacent unblurred frames. First, a motion-compensated predictor for the blurred frame is derived from its neighboring unblurred frame via specific motion estimation. Then, an accurate blur kernel, which is difficult to directly obtain from the blurred frame itself, is computed using both the predictor and the blurred frame. Next, a residual deconvolution is applied to both of those frames in order to reduce the ringing artifacts inherently caused by conventional deconvolution. The blur kernel estimation and deconvolution processes are iteratively performed for the deblurred frame. Simulation results show that the proposed algorithm provides superior deblurring results over conventional deblurring algorithms while preserving details and reducing ringing artifacts.

Semi-blind sparse image reconstruction with application to MRFM

We propose a solution to the image deconvolution problem where the convolution kernel or point spread function (PSF) is assumed to be only partially known. Small perturbations generated from the model are exploited to produce a few principal components explaining the PSF uncertainty in a high-dimensional space. Unlike recent developments on blind deconvolution of natural images, we assume the image is sparse in the pixel basis, a natural sparsity arising in magnetic resonance force microscopy (MRFM). Our approach adopts a Bayesian Metropolis-within-Gibbs sampling framework. The performance of our Bayesian semi-blind algorithm for sparse images is superior to previously proposed semi-blind algorithms such as the alternating minimization algorithm and blind algorithms developed for natural images. We illustrate our myopic algorithm on real MRFM tobacco virus data.