A unified approach to superresolution and multichannel blind deconvolution

Improved resolution in x-ray tomography by super-resolution

In this paper, super-resolution imaging is described and evaluated for x-ray tomography and is compared with standard tomography and upscaling during reconstruction. Blurring is minimized due to the negligible point spread of photon counting detectors and an electromagnetically movable micro-focus x-ray spot. Scans are acquired in high and low magnification geometry, where the latter is used to minimize penumbral blurring from the x-ray source. Sharpness and level of detail can be significantly increased in reconstructed slices to the point where the source size becomes the limiting factor. The achieved resolution of the different methods is quantified and compared using biological samples via the edge spread function, modulation transfer function, and Fourier ring correlation.

Modularized data-driven reconstruction framework for nonideal focal spot effect elimination in computed tomography

PURPOSE

High-performance computed tomography (CT) plays a vital role in clinical decision-making. However, the performance of CT imaging is adversely affected by the nonideal focal spot size of the x-ray source or degraded by an enlarged focal spot size due to aging. In this work, we aim to develop a deep learning-based strategy to mitigate the problem so that high spatial resolution CT images can be obtained even in the case of a nonideal x-ray source.

METHODS

To reconstruct high-quality CT images from blurred sinograms via joint image and sinogram learning, a cross-domain hybrid model is formulated via deep learning into a modularized data-driven reconstruction (MDR) framework. The proposed MDR framework comprises several blocks, and all the blocks share the same network architecture and network parameters. In essence, each block utilizes two sub-models to generate an estimated blur kernel and a high-quality CT image simultaneously. In this way, our framework generates not only a final high-quality CT image but also a series of intermediate images with gradually improved anatomical details, enhancing the visual perception for clinicians through the dynamic process. We used simulated training datasets to train our model in an end-to-end manner and tested our model on both simulated and realistic experimental datasets.

RESULTS

On the simulated testing datasets, our approach increases the information fidelity criterion (IFC) by up to 34.2%, the universal quality index (UQI) by up to 20.3%, the signal-to-noise (SNR) by up to 6.7%, and reduces the root mean square error (RMSE) by up to 10.5% as compared with FBP. Compared with the iterative deconvolution method (NSM), MDR increases IFC by up to 24.7%, UQI by up to 16.7%, SNR by up to 6.0%, and reduces RMSE by up to 9.4%. In the modulation transfer function (MTF) experiment, our method improves the MTF_50% by 34.5% and MTF_10% by 18.7% as compared with FBP, Similarly remarkably, our method improves MTF_50% by 14.3% and MTF_10% by 0.9% as compared with NSM. Also, our method shows better imaging results in the edge of bony structures and other tiny structures in the experiments using phantom consisting of ham and a bottle of peanuts.

CONCLUSIONS

A modularized data-driven CT reconstruction framework is established to mitigate the blurring effect caused by a nonideal x-ray source with relatively large focal spot. The proposed method enables us to obtain high-resolution images with less ideal x-ray source.

Blind deconvolution of sources of opportunity in ocean waveguides using bilinear channel models

A general blind deconvolution algorithmic framework is developed for sources of opportunity (e.g., ships at known locations) in an ocean waveguide. Here, both channel impulse responses (CIRs) and unknown source signals need to be simultaneously estimated from only the recorded signals on a receiver array using blind deconvolution, which is generally an ill-posed problem without any a priori information or additional assumptions about the underlying structure of the CIRs. By exploiting the typical ray-like arrival-time structure of the CIRs between a surface source and the elements of a vertical line array (VLA) in ocean waveguides, a principle component analysis technique is applied to build a bilinear parametric model linking the amplitudes and arrival-times of the CIRs across all channels for a variety of admissible ocean environments. The bilinear channel representation further reduces the dimension of the channel parametric model compared to linear models. A truncated power interaction deconvolution algorithm is then developed by applying the bilinear channel model to the traditional subspace deconvolution method. Numerical and experimental results demonstrate the robustness of this blind deconvolution methodology.

Super-resolution reconstruction of real infrared images acquired with unmanned aerial vehicle

Super-resolution (SR) technology provides a far promising computational imaging approach in obtaining a high-resolution (HR) image (or image sequences) from observed multiple low-resolution (LR) images by incorporating complementary information. In this paper, a three-stage SR method is proposed to generate a HR image from infrared (IR) LR Images acquired with Unmanned Aerial Vehicle (UAV). The proposed method integrates a high-level image capturing process and a low-level SR process. In this integrated process, we incorporate UAV path optimization, sub-pixel image registration, and sparseness constraint into a computational imaging framework of a region of interest (ROI). To refine ROI complementary feathers, we design an optimal flight control scheme to acquire adequate image sequences from multi-angles. In particular, a phase correlation approach achieving reliable sub-pixel image feature matching is adapted, on the basis of which an effective sparseness regularization model is built to enhance the fine structures of the IR image. Unlike most traditional multiple-frame SR algorithms that mainly focus on signal processing and achieve good performances when using standard test datasets, the performed experiments with real-life IR sequences indicate the three-stage SR method can also deal with practical LR IR image sequences collected by UAVs. The experimental results demonstrate that the proposed method is capable of generating HR images with good performance in terms of edge preservation and detail enhancement.

Enhancement of Acoustic Microscopy Lateral Resolution: A Comparison Between Deep Learning and Two Deconvolution Methods

Scanning acoustic microscopy (SAM) provides high-resolution images of biological tissues. Since higher transducer frequencies limit penetration depth, image resolution enhancement techniques could help in maintaining sufficient lateral resolution without sacrificing penetration depth. Compared with existing SAM research, this work introduces two novelties. First, deep learning (DL) is used to improve lateral resolution of 180-MHz SAM images, comparing it with two deconvolution-based approaches. Second, 316-MHz images are used as ground truth in order to quantitatively evaluate image resolution enhancement. The samples used were mouse and rat brain sections. The results demonstrate that DL can closely approximate ground truth (NRMSE = 0.056 and PSNR = 28.4 dB) even with a relatively limited training set (four images, each smaller than 1 mm ×1 mm). This study suggests the high potential of using DL as a single image superresolution method in SAM.

Multi-Image Blind Super-Resolution of 3D Scenes

We address the problem of estimating the latent high-resolution (HR) image of a 3D scene from a set of non-uniformly motion blurred low-resolution (LR) images captured in the burst mode using a hand-held camera. Existing blind super-resolution (SR) techniques that account for motion blur are restricted to fronto-parallel planar scenes. We initially develop an SR motion blur model to explain the image formation process in 3D scenes. We then use this model to solve for the three unknowns-the camera trajectories, the depth map of the scene, and the latent HR image. We first compute the global HR camera motion corresponding to each LR observation from patches lying on a reference depth layer in the input images. Using the estimated trajectories, we compute the latent HR image and the underlying depth map iteratively using an alternating minimization framework. Experiments on synthetic and real data reveal that our proposed method outperforms the state-of-the-art techniques by a significant margin.

PIZZARO: Forensic analysis and restoration of image and video data

This paper introduces a set of methods for image and video forensic analysis. They were designed to help to assess image and video credibility and origin and to restore and increase image quality by diminishing unwanted blur, noise, and other possible artifacts. The motivation came from the best practices used in the criminal investigation utilizing images and/or videos. The determination of the image source, the verification of the image content, and image restoration were identified as the most important issues of which automation can facilitate criminalists work. Novel theoretical results complemented with existing approaches (LCD re-capture detection and denoising) were implemented in the PIZZARO software tool, which consists of the image processing functionality as well as of reporting and archiving functions to ensure the repeatability of image analysis procedures and thus fulfills formal aspects of the image/video analysis work. Comparison of new proposed methods with the state of the art approaches is shown. Real use cases are presented, which illustrate the functionality of the developed methods and demonstrate their applicability in different situations. The use cases as well as the method design were solved in tight cooperation of scientists from the Institute of Criminalistics, National Drug Headquarters of the Criminal Police and Investigation Service of the Police of the Czech Republic, and image processing experts from the Czech Academy of Sciences.

Robust All-in-Focus Super-Resolution for Focal Stack Photography

We present an unconventional image super-resolution algorithm targeting focal stack images. Contrary to previous works, which align multiple images with sub-pixel accuracy for image super-resolution, we analyze the correlation among the differently focused narrow depth-of-field images in a focal stack to infer high-resolution details for image super-resolution. In order to accurately model the defocus kernels at different depths, we use a cubic interpolation to parameterize the projection of defocus kernels, and apply the radon transform to accurately reconstruct the defocus kernels at arbitrary depth. In the image super-resolution, we utilize the multi-image deconvolution method with a l1 -norm regularization to suppress noise and ringing artifacts. We have also extended the depth-of-field of our inputs to produce an all-in-focus super-resolution image. The effectiveness of our algorithm is demonstrated with the quantitative analysis using synthetic examples and the qualitative analysis using real-world examples.

Blind Super Resolution of Real-Life Video Sequences

Super resolution (SR) for real-life video sequences is a challenging problem due to complex nature of the motion fields. In this paper, a novel blind SR method is proposed to improve the spatial resolution of video sequences, while the overall point spread function of the imaging system, motion fields, and noise statistics are unknown. To estimate the blur(s), first, a nonuniform interpolation SR method is utilized to upsample the frames, and then, the blur(s) is(are) estimated through a multi-scale process. The blur estimation process is initially performed on a few emphasized edges and gradually on more edges as the iterations continue. Also for faster convergence, the blur is estimated in the filter domain rather than the pixel domain. The high-resolution frames are estimated using a cost function that has the fidelity and regularization terms of type Huber-Markov random field to preserve edges and fine details. The fidelity term is adaptively weighted at each iteration using a masking operation to suppress artifacts due to inaccurate motions. Very promising results are obtained for real-life videos containing detailed structures, complex motions, fast-moving objects, deformable regions, or severe brightness changes. The proposed method outperforms the state of the art in all performed experiments through both subjective and objective evaluations. The results are available online at http://lyle.smu.edu/~rajand/Video_SR/.

High spectral specificity of local chemical components characterization with multichannel shift-excitation Raman spectroscopy

Raman spectroscopy has emerged as a promising tool for its noninvasive and nondestructive characterization of local chemical structures. However, spectrally overlapping components prevent the specific identification of hyperfine molecular information of different substances, because of limitations in the spectral resolving power. The challenge is to find a way of preserving scattered photons and retrieving hidden/buried Raman signatures to take full advantage of its chemical specificity. Here, we demonstrate a multichannel acquisition framework based on shift-excitation and slit-modulation, followed by mathematical post-processing, which enables a significant improvement in the spectral specificity of Raman characterization. The present technique, termed shift-excitation blind super-resolution Raman spectroscopy (SEBSR), uses multiple degraded spectra to beat the dispersion-loss trade-off and facilitate high-resolution applications. It overcomes a fundamental problem that has previously plagued high-resolution Raman spectroscopy: fine spectral resolution requires large dispersion, which is accompanied by extreme optical loss. Applicability is demonstrated by the perfect recovery of fine structure of the C-Cl bending mode as well as the clear discrimination of different polymorphs of mannitol. Due to its enhanced discrimination capability, this method offers a feasible route at encouraging a broader range of applications in analytical chemistry, materials and biomedicine.

Image reconstruction from phased-array data based on multichannel blind deconvolution

In this paper we consider image reconstruction from fully sampled multichannel phased array MRI data without knowledge of the coil sensitivities. To overcome the non-uniformity of the conventional sum-of-square reconstruction, a new framework based on multichannel blind deconvolution (MBD) is developed for joint estimation of the image function and the sensitivity functions in image domain. The proposed approach addresses the non-uniqueness of the MBD problem by exploiting the smoothness of both functions in the image domain through regularization. Results using simulation, phantom and in vivo experiments demonstrate that the reconstructions by the proposed algorithm are more uniform than those by the existing methods.

Deskewing of underwater images

We address the problem of restoring a static planar scene degraded by skewing effect when imaged through adynamic water surface. In particular, we investigate geometric distortions due to unidirectional cyclic waves and circular ripples,phenomena that are most prevalent in fluid flow. Although the camera and scene are stationary, light rays emanating from a scene undergo refraction at the fluid–air interface. This refraction effect is time varying for dynamic fluids and results in nonrigid distortions (skew) in the captured image. These distortions can be associated with motion blur depending on the exposure time of the camera. In the first part of this paper, we establish the condition under which the blur induced due to unidirectional cyclic waves can be treated as space invariant. We proceed to derive a mathematical model for blur formation and propose a restoration scheme using a single degraded observation. In the second part, we reveal how the blur induced by circular ripples(though space variant) can be modeled as uniform in the polar domain and develop a method for deskewing. The proposed methods are tested on synthetic as well as real examples.

Optimal design of multichannel equalizers for the structural similarity index

The optimization of multichannel equalizers is studied for the structural similarity (SSIM) criteria. The closed-form formula is provided for the optimal equalizer when the mean of the source is zero. The formula shows that the equalizer with maximal SSIM index is equal to the one with minimal mean square error (MSE) multiplied by a positive real number, which is shown to be equal to the inverse of the achieved SSIM index. The relation of the maximal SSIM index to the minimal MSE is also established for given blurring filters and fixed length equalizers. An algorithm is also presented to compute the suboptimal equalizer for the general sources. Various numerical examples are given to demonstrate the effectiveness of the results.

Unified multiframe super-resolution of matte, foreground, and background

Reconstruction of a super-resolved image from multiple frames and extraction of matte are two popular topics that have been solved independently. In this paper, we advocate a unified framework that assimilates matting within the super-resolution model. We show that joint estimation is advantageous, as super-resolved edge information helps in obtaining a sharp matte, while the matte in turn aids in resolving fine details. We propose a multiframe approach to increase the spatial resolution of the matte, foreground, and background. This is validated extensively on examples from standard matting datasets.

Unified blind method for multi-image super-resolution and single/multi-image blur deconvolution

This paper presents, for the first time, a unified blind method for multi-image super-resolution (MISR or SR), single-image blur deconvolution (SIBD), and multi-image blur deconvolution (MIBD) of low-resolution (LR) images degraded by linear space-invariant (LSI) blur, aliasing, and additive white Gaussian noise (AWGN). The proposed approach is based on alternating minimization (AM) of a new cost function with respect to the unknown high-resolution (HR) image and blurs. The regularization term for the HR image is based upon the Huber-Markov random field (HMRF) model, which is a type of variational integral that exploits the piecewise smooth nature of the HR image. The blur estimation process is supported by an edge-emphasizing smoothing operation, which improves the quality of blur estimates by enhancing strong soft edges toward step edges, while filtering out weak structures. The parameters are updated gradually so that the number of salient edges used for blur estimation increases at each iteration. For better performance, the blur estimation is done in the filter domain rather than the pixel domain, i.e., using the gradients of the LR and HR images. The regularization term for the blur is Gaussian (L2 norm), which allows for fast noniterative optimization in the frequency domain. We accelerate the processing time of SR reconstruction by separating the upsampling and registration processes from the optimization procedure. Simulation results on both synthetic and real-life images (from a novel computational imager) confirm the robustness and effectiveness of the proposed method.

Commutability of blur and affine warping in super-resolution with application to joint estimation of triple-coupled variables

This paper proposes a new approach to the image blind super-resolution (BSR) problem in the case of affine interframe motion. Although the tasks of image registration, blur identification, and high-resolution (HR) image reconstruction are coupled in the imaging process, when dealing with nonisometric interframe motion or without the exact knowledge of the blurring process, classic SR techniques generally have to tackle them (maybe in some combinations) separately. The main difficulty is that state-of-the-art deconvolution methods cannot be straightforwardly generalized to cope with the space-variant motion. We prove that the operators of affine warping and blur commute with some additional transforms and derive an equivalent form of the BSR observation model. Using this equivalent form, we develop an iterative algorithm to jointly estimate the triple-coupled variables, i.e., the motion parameters, blur kernels, and HR image. Experiments on synthetic and real-life images illustrate the performance of the proposed technique in modeling the space-variant degradation process and restoring local textures.

Ambiguity and regularization in parallel MRI

In this paper, we formulate the parallel magnetic resonance imaging(pMRI) as a multichannel blind deconvolution problem with subsampling. First, the model allows formal characterization of image solutions consistent with data obtained by uniform subsampling of k-space. Second, the model allows analysis of the minimum set of required calibration data. Third, the filter bank formulation provides analysis of the sufficient sizes of interpolation kernels in widely used reconstruction techniques. Fourth, the model suggests principled development of regularization terms to fight ambiguity and ill-conditioning; specifically, subspace regularization is adapted from the blind image super-resolution work of Sroubek et al. [11]. Finally, characterization of the consistent set of image solutions leads to a cautionary comment on L1 regularization for the peculiar class of piece-wise constant images. Thus, it is proposed that the analysis of the subsampled blind deconvolution task provides insight into both the multiply determined nature of the pMRI task and possible design strategies for sampling and reconstruction.

Bayesian blind deconvolution from differently exposed image pairs

Photographs acquired under low-lighting conditions require long exposure times and therefore exhibit significant blurring due to the shaking of the camera. Using shorter exposure times results in sharper images but with a very high level of noise. In this paper we address the problem of utilizing two such images in order to obtain an estimate of the original scene and present a novel blind deconvolution algorithm for solving it. We formulate the problem in a hierarchical Bayesian framework by utilizing prior knowledge on the unknown image and blur, and also on the dependency between the two observed images. By incorporating a fully Bayesian analysis, the developed algorithm estimates all necessary model parameters along with the unknown image and blur, such that no user-intervention is needed. Moreover, we employ a variational Bayesian inference procedure, which allows for the statistical compensation of errors occurring at different stages of the restoration, and also provides uncertainties of the estimates. Experimental results with synthetic and real images demonstrate that the proposed method provides very high quality restoration results and compares favorably to existing methods even though no user supervision is needed.

Enhancement of coupled multichannel images using sparsity constraints

We consider the problem of joint enhancement of multichannel images with pixel based constraints on the multichannel data. Previous work by Cetin and Karl introduced nonquadratic regularization methods for SAR image enhancement using sparsity enforcing penalty terms. We formulate an optimization problem that jointly enhances complex-valued multichannel images while preserving the cross-channel information, which we include as constraints tying the multichannel images together. We pose this problem as a joint optimization problem with constraints. We first reformulate it as an equivalent (unconstrained) dual problem and develop a numerically-efficient method for solving it. We develop the Dual Descent method, which has low complexity, for solving the joint optimization problem. The algorithm is applied to both an interferometric synthetic aperture radar (IFSAR) problem, in which the relative phase between two complex-valued images indicate height, and to a synthetic multimodal medical image example.

Correction of spatially varying image and video motion blur using a hybrid camera

We describe a novel approach to reduce spatially varying motion blur in video and images using a hybrid camera system. A hybrid camera is a standard video camera that is coupled with an auxiliary low-resolution camera sharing the same optical path but capturing at a significantly higher frame rate. The auxiliary video is temporally sharper but at a lower resolution, while the lower frame-rate video has higher spatial resolution but is susceptible to motion blur. Our deblurring approach uses the data from these two video streams to reduce spatially varying motion blur in the high-resolution camera with a technique that combines both deconvolution and super-resolution. Our algorithm also incorporates a refinement of the spatially varying blur kernels to further improve results. Our approach can reduce motion blur from the high-resolution video as well as estimate new high-resolution frames at a higher frame rate. Experimental results on a variety of inputs demonstrate notable improvement over current state-of-the-art methods in image/video deblurring.

Statistical performance modeling for superresolution: a discrete data-continuous reconstruction framework

We address performance modeling of superresolution (SR) techniques. Superresolution consists in combining several images of the same scene to produce an image with better resolution and contrast. We propose a discrete data-continuous reconstruction framework to conduct SR performance analysis and derive a theoretical expression of the reconstruction mean squared error (MSE) as a compact, computationally tractable function of signal-to-noise ratio (SNR), scene model, sensor transfer function, number of frames, interframe translation motion, and SR reconstruction filter. A formal expression for the MSE is obtained that allows a qualitative study of SR behavior. In particular we provide an original outlook on the balance between noise and aliasing reduction in linear SR. Explicit account for the SR reconstruction filter is an original feature of our model. It allows for the first time to study not only optimal filters but also suboptimal ones, which are often used in practice.