A multi-faceted adaptive image fusion algorithm using a multi-wavelet-based matching measure in the PCNN domain

MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion

The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.

Infrared and visible image fusion using two-layer generative adversarial network

Infrared (IR) images can distinguish targets from their backgrounds based on difference in thermal radiation, whereas visible images can provide texture details with high spatial resolution. The fusion of the IR and visible images has many advantages and can be applied to applications such as target detection and recognition. This paper proposes a two-layer generative adversarial network (GAN) to fuse these two types of images. In the first layer, the network generate fused images using two GANs: one uses the IR image as input and the visible image as ground truth, and the other with the visible as input and the IR as ground truth. In the second layer, the network transfer one of the two fused images generated in the first layer as input and the other as ground truth to GAN to generate the final fused image. We adopt TNO and INO data sets to verify our method, and by comparing with eight objective evaluation parameters obtained by other ten methods. It is demonstrated that our method is able to achieve better performance than state-of-arts on preserving both texture details and thermal information.

PCNN Mechanism and its Parameter Settings

The pulse-coupled neural network (PCNN) model is a third-generation artificial neural network without training that uses the synchronous pulse bursts of neurons to process digital images, but the lack of in-depth theoretical research limits its extensive application. By analyzing the working mechanism of the PCNN, we present an expression for the fire-extinguishing time of neurons that fire in the second iteration and an expression for the firing time of neurons that extinguish in the second iteration. In addition, we find a phenomenon of the PCNN and name it mathematically coupled fire extinguishing. Based on the above analysis, we propose a new working mode for the PCNN, where the refiring of fire-extinguishing neurons is only allowed when all firing neurons are extinguished. We also work out the constraint conditions of the parameter settings under this mode. Furthermore, we analyze the relationship between the network parameters and mathematically coupled fire extinguishing, the coupling of neighboring neurons, and the convergence rate of the PCNN, respectively. In addition, we demonstrate the essential regularity of extinguished neuron in the PCNN and then propose an optimal parameter setting to achieve the best comprehensive performance of the PCNN.