An optimal quadtree-based motion estimation and motion-compensated interpolation scheme for video compression

A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication Channel

We present a novel adaptive multimodal intensity-event algorithm to optimize an overall objective of object tracking under bit rate constraints for a host-chip architecture. The chip is a computationally resource-constrained device acquiring high-resolution intensity frames and events, while the host is capable of performing computationally expensive tasks. We develop a joint intensity-neuromorphic event rate-distortion compression framework with a quadtree (QT)-based compression of intensity and events scheme. The goal of this compression framework is to optimally allocate bits to the intensity frames and neuromorphic events based on the minimum distortion at a given communication channel capacity. The data acquisition on the chip is driven by the presence of objects of interest in the scene as detected by an object detector. The most informative intensity and event data are communicated to the host under rate constraints so that the best possible tracking performance is obtained. The detection and tracking of objects in the scene are done on the distorted data at the host. Intensity and events are jointly used in a fusion framework to enhance the quality of the distorted images, in order to improve the object detection and tracking performance. The performance assessment of the overall system is done in terms of the multiple object tracking accuracy (MOTA) score. Compared with using intensity modality only, there is an improvement in MOTA using both these modalities in different scenarios.

Content Adaptive Lagrange Multiplier Selection for Rate-Distortion Optimization in 3-D Wavelet-Based Scalable Video Coding

Rate-distortion optimization (RDO) plays an essential role in substantially enhancing the coding efficiency. Currently, rate-distortion optimized mode decision is widely used in scalable video coding (SVC). Among all the possible coding modes, it aims to select the one which has the best trade-off between bitrate and compression distortion. Specifically, this tradeoff is tuned through the choice of the Lagrange multiplier. Despite the prevalence of conventional method for Lagrange multiplier selection in hybrid video coding, the underlying formulation is not applicable to 3-D wavelet-based SVC where the explicit values of the quantization step are not available, with on consideration of the content features of input signal. In this paper, an efficient content adaptive Lagrange multiplier selection algorithm is proposed in the context of RDO for 3-D wavelet-based SVC targeting quality scalability. Our contributions are two-fold. First, we introduce a novel weighting method, which takes account of the mutual information, gradient per pixel, and texture homogeneity to measure the temporal subband characteristics after applying the motion-compensated temporal filtering (MCTF) technique. Second, based on the proposed subband weighting factor model, we derive the optimal Lagrange multiplier. Experimental results demonstrate that the proposed algorithm enables more satisfactory video quality with negligible additional computational complexity.

Wavelet-based joint estimation and encoding of depth-image-based representations for free-viewpoint rendering

We propose a wavelet-based codec for the static depth-image-based representation, which allows viewers to freely choose the viewpoint. The proposed codec jointly estimates and encodes the unknown depth map from multiple views using a novel rate-distortion (RD) optimization scheme. The rate constraint reduces the ambiguity of depth estimation by favoring piecewise-smooth depth maps. The optimization is efficiently solved by a novel dynamic programming along trees of integer wavelet coefficients. The codec encodes the image and the depth map jointly to decrease their redundancy and to provide a RD-optimized bitrate allocation between the two. The codec also offers scalability both in resolution and in quality. Experiments on real data show the effectiveness of the proposed codec.

Video compression with dense motion fields

We propose a motion-compensated video coding system employing dense motion fields. The dense motion field is calculated at the transmitter, and the motion information is efficiently encoded and transmitted along with the residual frame. The motion estimation is performed by existing techniques in the literature, while we focus on the coding of the motion field and the displaced frame difference (DFD) frame. The dense motion field formulation leads to several novel and distinct advantages. The motion field is encoded in a lossy manner to make the motion rate manageable. The more accurate and precise motion description allows us to predict where the DFD energy will be significant, thus leading to a more efficient DFD encoder compared to applying traditional still-image coding techniques. Furthermore, the dense motion field framework allows us to refine and tailor the motion estimation process such that the resulting DFD frame is easier to encode. Simulations demonstrate superior performance against standard block-based coders, with greater advantages for sequences with more complex motion.