Anatomical-functional image fusion by information of interest in local Laplacian filtering domain

Continuum topological derivative - a novel application tool for denoising CT and MRI medical images

BACKGROUND

CT and MRI modalities are important diagnostics tools for exploring the anatomical and tissue properties, respectively of the human beings. Several advancements like HRCT, FLAIR and Propeller have advantages in diagnosing the diseases very accurately, but still have enough space for improvements due to the presence of inherent and instrument noises. In the case of CT and MRI, the quantum mottle and the Gaussian and Rayleigh noises, respectively are still present in their advanced modalities of imaging. This paper addresses the denoising problem with continuum topological derivative technique and proved its trustworthiness based on the comparative study with other traditional filtration methods such as spatial, adaptive, frequency and transformation techniques using measures like visual inspection and performance metrics.

METHODS

This research study focuses on identifying a novel method for denoising by testing different filters on HRCT (High-Resolution Computed Tomography) and MR (Magnetic Resonance) images. The images were acquired from the Image Art Radiological Scan Centre using the SOMATOM CT and SIGNA Explorer (operating at 1.5 Tesla) machines. To compare the performance of the proposed CTD (Continuum Topological Derivative) method, various filters were tested on both HRCT and MR images. The filters tested for comparison were Gaussian (2D convolution operator), Wiener (deconvolution operator), Laplacian and Laplacian diagonal (2nd order partial differential operator), Average, Minimum, and Median (ordinary spatial operators), PMAD (Anisotropic diffusion operator), Kuan (statistical operator), Frost (exponential convolution operator), and HAAR Wavelet (time-frequency operator). The purpose of the study was to evaluate the effectiveness of the CTD method in removing noise compared to the other filters. The performance metrics were analyzed to assess the diligence of noise removal achieved by the CTD method. The primary outcome of the study was the removal of quantum mottle noise in HRCT images, while the secondary outcome focused on removing Gaussian (foreground) and Rayleigh (background) noise in MR images. The study aimed to observe the dynamics of noise removal by examining the values of the performance metrics. In summary, this study aimed to assess the denoising ability of various filters in HRCT and MR images, with the CTD method being the proposed approach. The study evaluated the performance of each filter using specific metrics and compared the results to determine the effectiveness of the CTD method in removing noise from the images.

RESULTS

Based on the calculated performance metric values, it has been observed that the CTD method successfully removed quantum mottle noise in HRCT images and Gaussian as well as Rayleigh noise in MRI. This can be evidenced by the PSNR (Peak Signal-to-Noise Ratio) metric, which consistently exhibited values ranging from 50 to 65 for all the tested images. Additionally, the CTD method demonstrated remarkably low residual values, typically on the order of e-09, which is a distinctive characteristic across all the images. Furthermore, the performance metrics of the CTD method consistently outperformed those of the other tested methods. Consequently, the results of this study have significant implications for the quality, structural similarity, and contrast of HRCT and MR images, enabling clinicians to obtain finer details for diagnostic purposes.

CONCLUSION

Continuum topological derivative algorithm is found to be constructive in removing prominent noises in both CT and MRI images and can serve as a potential tool for recognition of anatomical details in case of diseased and normal ones. The results obtained from this research work are highly inspiring and offer great promise in obtaining accurate diagnostic information for critical cases such as Thoracic Cavity Carina, Brain SPI Globe Lens 4th Ventricle, Brain-Middle Cerebral Artery, Brain-Middle Cerebral Artery and neoplastic lesions. These findings lay the foundation for implementing the proposed CTD technique in routine clinical diagnosis.

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.

Unsupervised Fusion of Misaligned PAT and MRI Images via Mutually Reinforcing Cross-Modality Image Generation and Registration

Photoacoustic tomography (PAT) and magnetic resonance imaging (MRI) are two advanced imaging techniques widely used in pre-clinical research. PAT has high optical contrast and deep imaging range but poor soft tissue contrast, whereas MRI provides excellent soft tissue information but poor temporal resolution. Despite recent advances in medical image fusion with pre-aligned multimodal data, PAT-MRI image fusion remains challenging due to misaligned images and spatial distortion. To address these issues, we propose an unsupervised multi-stage deep learning framework called PAMRFuse for misaligned PAT and MRI image fusion. PAMRFuse comprises a multimodal to unimodal registration network to accurately align the input PAT-MRI image pairs and a self-attentive fusion network that selects information-rich features for fusion. We employ an end-to-end mutually reinforcing mode in our registration network, which enables joint optimization of cross-modality image generation and registration. To the best of our knowledge, this is the first attempt at information fusion for misaligned PAT and MRI. Qualitative and quantitative experimental results show the excellent performance of our method in fusing PAT-MRI images of small animals captured from commercial imaging systems.

Segmentation-Based Fusion of CT and MR Images

In this paper, a segmentation-based image fusion method is proposed for the fusion of MR and CT images to obtain a high contrast fused image that contains complementary information from both input images. The proposed method uses the fuzzy C-mean method to extract information about the skull from the CT image. This skull information is used to extract soft tissue information from the MR image. Both the skull information and the soft tissue information are then fused using the fusion rule. The efficiency of the proposed method over other state-of-the-art fusion methods is analyzed and compared using qualitative and quantitative analysis methods. Qualitative analysis shows the improvement in the contrast between the bone and the soft tissue using the proposed method over other state-of-the-art methods without introducing any artifacts or distortions. Classical and gradient-based quantitative analysis also show significant improvement in the fused image obtained using the proposed method over the five state-of-the-art methods. The percentage improvement in the standard deviation, average gradient, entropy, spatial frequency, QABF, and LABF of the proposed method over the best value obtained by the five state-of-the-art methods is 27.11%, 12.06%, 23.64%, 11.30%, 5.59%, and 13.70% respectively.

A novel medical image fusion method based on multi-scale shearing rolling weighted guided image filter

Medical image fusion is a crucial technology for biomedical diagnoses. However, current fusion methods struggle to balance algorithm design, visual effects, and computational efficiency. To address these challenges, we introduce a novel medical image fusion method based on the multi-scale shearing rolling weighted guided image filter (MSRWGIF). Inspired by the rolling guided filter, we construct the rolling weighted guided image filter (RWGIF) based on the weighted guided image filter. This filter offers progressive smoothing filtering of the image, generating smooth and detailed images. Then, we construct a novel image decomposition tool, MSRWGIF, by replacing non-subsampled shearlet transform's non-sampling pyramid filter with RWGIF to extract richer detailed information. In the first step of our method, we decompose the original images under MSRWGIF to obtain low-frequency subbands (LFS) and high-frequency subbands (HFS). Since LFS contain a large amount of energy-based information, we propose an improved local energy maximum (ILGM) fusion strategy. Meanwhile, HFS employ a fast and efficient parametric adaptive pulse coupled-neural network (AP-PCNN) model to combine more detailed information. Finally, the inverse MSRWGIF is utilized to generate the final fused image from fused LFS and HFS. To test the proposed method, we select multiple medical image sets for experimental simulation and confirm its advantages by combining seven high-quality representative metrics. The simplicity and efficiency of the method are compared with 11 classical fusion methods, illustrating significant improvements in the subjective and objective performance, especially for color medical image fusion.

Multi-Sensor Medical-Image Fusion Technique Based on Embedding Bilateral Filter in Least Squares and Salient Detection

A multi-sensor medical-image fusion technique, which integrates useful information from different single-modal images of the same tissue and provides a fused image that is more comprehensive and objective than a single-source image, is becoming an increasingly important technique in clinical diagnosis and treatment planning. The salient information in medical images often visually describes the tissue. To effectively embed salient information in the fused image, a multi-sensor medical image fusion method is proposed based on an embedding bilateral filter in least squares and salient detection via a deformed smoothness constraint. First, source images are decomposed into base and detail layers using a bilateral filter in least squares. Then, the detail layers are treated as superpositions of salient regions and background information; a fusion rule for this layer based on the deformed smoothness constraint and guided filtering was designed to successfully conserve the salient structure and detail information of the source images. A base-layer fusion rule based on modified Laplace energy and local energy is proposed to preserve the energy information of these source images. The experimental results demonstrate that the proposed method outperformed nine state-of-the-art methods in both subjective and objective quality assessments on the Harvard Medical School dataset.

A multimodal fusion method for Alzheimer's disease based on DCT convolutional sparse representation

Introduction

The medical information contained in magnetic resonance imaging (MRI) and positron emission tomography (PET) has driven the development of intelligent diagnosis of Alzheimer's disease (AD) and multimodal medical imaging. To solve the problems of severe energy loss, low contrast of fused images and spatial inconsistency in the traditional multimodal medical image fusion methods based on sparse representation. A multimodal fusion algorithm for Alzheimer' s disease based on the discrete cosine transform (DCT) convolutional sparse representation is proposed.

Methods

The algorithm first performs a multi-scale DCT decomposition of the source medical images and uses the sub-images of different scales as training images, respectively. Different sparse coefficients are obtained by optimally solving the sub-dictionaries at different scales using alternating directional multiplication method (ADMM). Secondly, the coefficients of high-frequency and low-frequency subimages are inverse DCTed using an improved L1 parametric rule combined with improved spatial frequency novel sum-modified SF (NMSF) to obtain the final fused images.

Results and discussion

Through extensive experimental results, we show that our proposed method has good performance in contrast enhancement, texture and contour information retention.

Non-uniform illumination underwater image enhancement via events and frame fusion

Absorption and scattering by aqueous media can attenuate light and cause underwater optical imagery difficulty. Artificial light sources are usually used to aid deep-sea imaging. Due to the limited dynamic range of standard cameras, artificial light sources often cause underwater images to be underexposed or overexposed. By contrast, event cameras have a high dynamic range and high temporal resolution but cannot provide frames with rich color characteristics. In this paper, we exploit the complementarity of the two types of cameras to propose an efficient yet simple method for image enhancement of uneven underwater illumination, which can generate enhanced images containing better scene details and colors similar to standard frames. Additionally, we create a dataset recorded by the Dynamic and Active-pixel Vision Sensor that includes both event streams and frames, enabling testing of the proposed method and frame-based image enhancement methods. The experimental results conducted on our dataset with qualitative and quantitative measures demonstrate that the proposed method outperforms the compared enhancement algorithms.

Medical image fusion quality assessment based on conditional generative adversarial network

Multimodal medical image fusion (MMIF) has been proven to effectively improve the efficiency of disease diagnosis and treatment. However, few works have explored dedicated evaluation methods for MMIF. This paper proposes a novel quality assessment method for MMIF based on the conditional generative adversarial networks. First, with the mean opinion scores (MOS) as the guiding condition, the feature information of the two source images is extracted separately through the dual channel encoder-decoder. The features of different levels in the encoder-decoder are hierarchically input into the self-attention feature block, which is a fusion strategy for self-identifying favorable features. Then, the discriminator is used to improve the fusion objective of the generator. Finally, we calculate the structural similarity index between the fake image and the true image, and the MOS corresponding to the maximum result will be used as the final assessment result of the fused image quality. Based on the established MMIF database, the proposed method achieves the state-of-the-art performance among the comparison methods, with excellent agreement with subjective evaluations, indicating that the method is effective in the quality assessment of medical fusion images.

MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer

Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

Multimodal fusion of tomographic sequences of medical images: MRE spatially enhanced by MRI

BACKGROUND AND OBJECTIVE

In biomedical fields, image analysis is often necessary for an accurate diagnosis. In order to obtain all the information needed to form an in-depth clinical picture, it may be useful to combine the contents of images taken under different diagnostic modes. Multimodal medical image fusion techniques enable complementary information acquired by different imaging devices to be automatically combined into a unique image.

METHODS

In this paper, multimodal medical images fusion method based on multiresolution analysis (MRA) is proposed, with the aim to combine the high geometric content of magnetic resonance imaging (MRI) and the elasticity information of magnetic resonance elastography (MRE), simultaneously acquired on the same organs of a patient. First, the slices of MRE are volumetrically interpolated to exactly overlap, each with a slice of MRI. Then, the spatial details of MRI are extracted by means of MRA and injected into the corresponding slices of MRE. Due to the intrinsic dissimilarity between corresponding slices of MRE and MRI, the spatial details of MRI are modulated by local or global matching functions.

RESULTS

The performance of the proposed method is quantitatively assessed considering radiometric and geometric consistency of the fused images with respect to their originals, in a comparison with two popular methods from the literature. For a qualitative evaluation, a visual inspection is carried out.

CONCLUSIONS

The results show that the proposed method enables an effective MRI-MRE fusion that allows the elasticity information and geometric details of the examined organs to be evaluated in a single image.

A Review on the Rule-Based Filtering Structure with Applications on Computational Biomedical Images

In this paper, we present rule-based fuzzy inference systems that consist of a series of mathematical representations based on fuzzy concepts in the filtering structure. It is crucial for understanding and discussing different principles associated with fuzzy filter design procedures. A number of typical fuzzy multichannel filtering approaches are provided in order to clarify the different fuzzy filter designs and compare different algorithms. In particular, in most practical applications (i.e., biomedical image analysis), the emphasis is placed primarily on fuzzy filtering algorithms, with the main advantages of restoration of corrupted medical images and the interpretation capability, along with the capability of edge preservation and relevant image information for accurate diagnosis of diseases.

Medical image fusion using fuzzy adaptive reduced pulse coupled neural networks

This paper addresses a novel neuro-fuzzy-based approach to set the weighted linking strength of parameter - adaptive reduced pulse coupled neural networks. In reduced PCNN based medical image fusion algorithms, it is quite essential to evaluate the prominence of each pixel in an image. The fusion performance in turn depends on the linking factor, internal activity. Thus, we need to set these values of reduced PCNN in a more adaptive manner with fewer complications and uncertainties. For this, the weighted linking strength i.e., lambda of the reduced PCNN neurons is attentively set by a fuzzy-based approach. Here, lambda of neurons is represented as fuzzy membership values using the activity level measures such as local information entropy and energy. Finally, a new model called-Fuzzy adaptive reduced pulse coupled neural networks is developed by reducing the number of parameters and fuzzy adaptive settings of them. This leads to a very less complicated network and more computational efficacy, which is a prominent part of health care requirements. The proposed scheme is free from the shortcomings such as loss of boundaries, structural details, unwanted artifacts, degradations, etc. Subjective and objective evaluations show better performance of this new approach compared to the existing techniques.

Multimodal medical image fusion using adaptive co-occurrence filter-based decomposition optimization model

MOTIVATION

Medical image fusion has developed into an important technology, which can effectively merge the significant information of multiple source images into one image. Fused images with abundant and complementary information are desirable, which contributes to clinical diagnosis and surgical planning.

RESULTS

In this article, the concept of the skewness of pixel intensity (SPI) and a novel adaptive co-occurrence filter (ACOF)-based image decomposition optimization model are proposed to improve the quality of fused images. Experimental results demonstrate that the proposed method outperforms 22 state-of-the-art medical image fusion methods in terms of five objective indices and subjective evaluation, and it has higher computational efficiency.

AVAILABILITY AND IMPLEMENTATION

First, the concept of SPI is applied to the co-occurrence filter to design ACOF. The initial base layers of source images are obtained using ACOF, which relies on the contents of images rather than fixed scale. Then, the widely used iterative filter framework is replaced with an optimization model to ensure that the base layer and detail layer are sufficiently separated and the image decomposition has higher computational efficiency. The optimization function is constructed based on the characteristics of the ideal base layer. Finally, the fused images are generated by designed fusion rules and linear addition. The code and data can be downloaded at https://github.com/zhunui/acof.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

A multiscale double-branch residual attention network for anatomical-functional medical image fusion

Medical image fusion technology synthesizes complementary information from multimodal medical images. This technology is playing an increasingly important role in clinical applications. In this paper, we propose a new convolutional neural network, which is called the multiscale double-branch residual attention (MSDRA) network, for fusing anatomical-functional medical images. Our network contains a feature extraction module, a feature fusion module and an image reconstruction module. In the feature extraction module, we use three identical MSDRA blocks in series to extract image features. The MSDRA block has two branches. The first branch uses a multiscale mechanism to extract features of different scales with three convolution kernels of different sizes, while the second branch uses six 3 × 3 convolutional kernels. In addition, we propose the Feature L₁-Norm fusion strategy to fuse the features obtained from the input images. Compared with the reference image fusion algorithms, MSDRA consumes less fusion time and achieves better results in visual quality and the objective metrics of Spatial Frequency (SF), Average Gradient (AG), Edge Intensity (EI), Quality-Aware Clustering (QAC), Variance (VAR), and Visual Information Fidelity for Fusion (VIFF).

Multimodal medical image fusion based on content-based decomposition and PCA-Sigmoid

OBJECTIVE

The objective of any multimodal medical image fusion algorithm is to assist a radiologist for better decision-making during the diagnosis and therapy by integrating the anatomical (magnetic resonance imaging) and functional (positron emission tomography/single-photon emission computed tomography) information.

METHODS

We proposed a new medical image fusion method based on content-based decomposition, principal component analysis (PCA), and sigmoid function. We considered empirical wavelet transform (EWT) for content-based decomposition purposes since it can preserve crucial medical image information such as edges and corners. PCA is used to obtain initial weights corresponding to each detail layer.

RESULTS

In our experiments, we found that direct usage of PCA for detail layer fusion introduces severe artifacts into the fused image due to weight scaling issues. In order to tackle this, we considered using the sigmoid function for better weight scaling. We considered 24 pairs of MRI-PET and 24 pairs of MRI-SPECT images for fusion and the results are measured using four significant quantitative metrics.

CONCLUSION

Finally, we compared our proposed method with other state-of-the-art transform-based fusion approaches, using traditional and recent performance measures. An appreciable improvement is observed in both qualitative and quantitative results compared to other fusion methods.

DMC-Fusion: Deep Multi-Cascade Fusion With Classifier-Based Feature Synthesis for Medical Multi-Modal Images

Multi-modal medical image fusion is a challenging yet important task for precision diagnosis and surgical planning in clinical practice. Although single feature fusion strategy such as Densefuse has achieved inspiring performance, it tends to be not fully preserved for the source image features. In this paper, a deep multi-fusion framework with classifier-based feature synthesis is proposed to automatically fuse multi-modal medical images. It consists of a pre-trained autoencoder based on dense connections, a feature classifier and a multi-cascade fusion decoder with separately fusing high-frequency and low-frequency. The encoder and decoder are transferred from MS-COCO datasets and pre-trained simultaneously on multi-modal medical image public datasets to extract features. The feature classification is conducted through Gaussian high-pass filtering and the peak signal to noise ratio thresholding, then feature maps in each layer of the pre-trained Dense-Block and decoder are divided into high-frequency and low-frequency sequences. Specifically, in proposed feature fusion block, parameter-adaptive pulse coupled neural network and l₁-weighted are employed to fuse high-frequency and low-frequency, respectively. Finally, we design a novel multi-cascade fusion decoder on total decoding feature stage to selectively fuse useful information from different modalities. We also validate our approach for the brain disease classification using the fused images, and a statistical significance test is performed to illustrate that the improvement in classification performance is due to the fusion. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance in both qualitative and quantitative evaluations.

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Multimodal medical imaging is an indispensable requirement in the treatment of various pathologies to accelerate care. Rather than discrete images, a composite image combining complementary features from multimodal images is highly informative for clinical examinations, surgical planning, and progress monitoring. In this paper, a deep learning fusion model is proposed for the fusion of medical multimodal images. Based on pyramidal and residual learning units, the proposed model, strengthened with adaptive fusion rules, is tested on image pairs from a standard dataset. The potential of the proposed model for enhanced image exams is shown by fusion studies with deep network images and quantitative output metrics of magnetic resonance imaging and positron emission tomography (MRI/PET) and magnetic resonance imaging and single-photon emission computed tomography (MRI/SPECT). The proposed fusion model achieves the Structural Similarity Index Measure (SSIM) values of 0.9502 and 0.8103 for the MRI/SPECT and MRI/PET MRI/SPECT image sets, signifying the perceptual visual consistency of the fused images. Testing is performed on 20 pairs of MRI/SPECT and MRI/PET images. Similarly, the Mutual Information (MI) values of 2.7455 and 2.7776 obtained for the MRI/SPECT and MRI/PET image sets, indicating the model’s ability to capture the information content from the source images to the composite image. Further, the proposed model allows deploying its variants, introducing refinements on the basic model suitable for the fusion of low and high-resolution medical images of diverse modalities.

Multi-modal Medical Image Fusion Algorithm Based on Spatial Frequency Motivated PA-PCNN in the NSST Domain

BACKGROUND

Image fusion has been grown as an effectual method in diseases related diagnosis schemes.

METHODS

In this paper, a new method for combining multimodal medical images using spatial frequency motivated parameter-adaptive PCNN (SF-PAPCNN) is suggested. The multi- modal images are disintegrated into frequency bands by using decomposition NSST. The coefficients of low frequency bands are selected using maximum rule. The coefficients of high frequency bands are combined by SF-PAPCNN.

METHODS

In this paper, a new method for combining multimodal medical images using spatial frequency motivated parameter-adaptive PCNN (SF-PAPCNN) is suggested. The multi-modal images are disintegrated into frequency bands by using decomposition NSST. The coefficients of low frequency bands are selected using maximum rule. The coefficients of high frequency bands are combined by SF-PAPCNN.

RESULTS

The fused medical images is obtained by applying INSST to above coefficients.

CONCLUSION

The quality metrics such as entropy ENT, fusion symmetry FS, deviation STD, mutual information Q_MI and edge strength Q_AB/F are used to validate the efficacy of suggested scheme.

Medical image fusion based on hybrid three-layer decomposition model and nuclear norm

The aim of medical image fusion technology is to synthesize multiple-image information to assist doctors in making scientific decisions. Existing studies have focused on preserving image details while avoiding halo artifacts and color distortions. This paper proposes a novel medical image fusion algorithm based on this research objective. First, the input image is decomposed into structure, texture, and local mean brightness layers using a hybrid three-layer decomposition model that can fully extract the features of the original images without the introduction of artifacts. Secondly, the nuclear norm of the patches, which are obtained using a sliding window, are calculated to construct the weight maps of the structure and texture layers. The weight map of the local mean brightness layer is constructed by calculating the local energy. Finally, remapping functions are applied to enhance each fusion layer, which reconstructs the final fusion image with the inverse operation of decomposition. Subjective and objective experiments confirm that the proposed algorithm has a distinct advantage compared with other state-of-the-art algorithms.

An adaptive two-scale biomedical image fusion method with statistical comparisons

Two-scale image representation of base and detail in the spatial-domain is a well-known decomposition scheme for its lower computational complexity than that performed in the transform-domain in the field of image fusion. Unfortunately, for a pseudo-colour input image, the base and detail images in the spatial-domain obtained via image decomposition scheme always display in greyscale. In this paper, a two-scale image fusion method with adaptive threshold obtained by Otsu's method is proposed for pseudo-colour image in the colour space domain. For greyscale image, detail and base image are obtained using structural information extracted from the difference image between a global and a local patch size. Consequently, local edge-preserving filter for preserving luminance information and local energy with the discussed window size are adopted to combine base and detail image. Experimental results show that structural and luminance information has been better preserved in terms of subjective and objective evaluations for medical image and protein image fusion. Specially, a two-step non-parametric statistical test (Friedman test and Nemenyi post-hoc test) with p-values is adopted to analysis the statistical significant of the relative difference between the proposed and compared methods in terms of values of objective metrics including 30 co-registered pairs of imaging data.

Multimodal medical image fusion via laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy

BACKGROUND

In recent years, numerous fusion algorithms have been proposed for multimodal medical images. The Laplacian pyramid is one type of multiscale fusion method. Although the pyramid-based fusion algorithm can fuse images well, it has the disadvantages of edge degradation, detail loss and image smoothing as the number of decomposition layers increase, which is harmful for medical diagnosis and analysis.

METHOD

This paper proposes a medical image fusion algorithm based on the Laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy, which can greatly improve the edge quality. First, multimodal medical images are reconstructed through convolutional neural network. Then, the Laplacian pyramid is applied in the decomposition and fusion process. The optimal number of decomposition layers is determined by experiments. In addition, a local gradient energy fusion strategy is utilized to fuse the coefficients in each layer. Finally, the fused image is output through Laplacian inverse transformation.

RESULTS

Compared with existing algorithms, our fusion results represent better vision quality performance. Furthermore, our algorithm is considerably superior to the compared algorithms in objective indicators. In addition, in our fusion results of Alzheimer and Glioma, the disease details are much clearer than those of compared algorithms, which can provide a reliable basis for doctors to analyze disease and make pathological diagnoses.

Multimodal medical image fusion algorithm in the era of big data

In image-based medical decision-making, different modalities of medical images of a given organ of a patient are captured. Each of these images will represent a modality that will render the examined organ differently, leading to different observations of a given phenomenon (such as stroke). The accurate analysis of each of these modalities promotes the detection of more appropriate medical decisions. Multimodal medical imaging is a research field that consists in the development of robust algorithms that can enable the fusion of image information acquired by different sets of modalities. In this paper, a novel multimodal medical image fusion algorithm is proposed for a wide range of medical diagnostic problems. It is based on the application of a boundary measured pulse-coupled neural network fusion strategy and an energy attribute fusion strategy in a non-subsampled shearlet transform domain. Our algorithm was validated in dataset with modalities of several diseases, namely glioma, Alzheimer’s, and metastatic bronchogenic carcinoma, which contain more than 100 image pairs. Qualitative and quantitative evaluation verifies that the proposed algorithm outperforms most of the current algorithms, providing important ideas for medical diagnosis.

Unsupervised Deep Image Fusion with Structure Tensor Representations

Convolutional neural networks (CNNs) have facilitated substantial progress on various problems in computer vision and image processing. However, applying them to image fusion has remained challenging due to the lack of the labelled data for supervised learning. This paper introduces a deep image fusion network (DIF-Net), an unsupervised deep learning framework for image fusion. The DIF-Net parameterizes the entire processes of image fusion, comprising of feature extraction, feature fusion, and image reconstruction, using a CNN. The purpose of DIF-Net is to generate an output image which has an identical contrast to high-dimensional input images. To realize this, we propose an unsupervised loss function using the structure tensor representation of the multi-channel image contrasts. Different from traditional fusion methods that involve time-consuming optimization or iterative procedures to obtain the results, our loss function is minimized by a stochastic deep learning solver with large-scale examples. Consequently, the proposed method can produce fused images that preserve source image details through a single forward network trained without reference ground-truth labels. The proposed method has broad applicability to various image fusion problems, including multi-spectral, multi-focus, and multi-exposure image fusions. Quantitative and qualitative evaluations show that the proposed technique outperforms existing state-of-the-art approaches for various applications.

Green Fluorescent Protein and Phase-Contrast Image Fusion via Generative Adversarial Networks

In the field of cell and molecular biology, green fluorescent protein (GFP) images provide functional information embodying the molecular distribution of biological cells while phase-contrast images maintain structural information with high resolution. Fusion of GFP and phase-contrast images is of high significance to the study of subcellular localization, protein functional analysis, and genetic expression. This paper proposes a novel algorithm to fuse these two types of biological images via generative adversarial networks (GANs) by carefully taking their own characteristics into account. The fusion problem is modelled as an adversarial game between a generator and a discriminator. The generator aims to create a fused image that well extracts the functional information from the GFP image and the structural information from the phase-contrast image at the same time. The target of the discriminator is to further improve the overall similarity between the fused image and the phase-contrast image. Experimental results demonstrate that the proposed method can outperform several representative and state-of-the-art image fusion methods in terms of both visual quality and objective evaluation.

An Adaptive Exposure Fusion Method Using fuzzy Logic and Multivariate Normal Conditional Random Fields

High dynamic range (HDR) has wide applications involving intelligent vision sensing which includes enhanced electronic imaging, smart surveillance, self-driving cars, intelligent medical diagnosis, etc. Exposure fusion is an essential HDR technique which fuses different exposures of the same scene into an HDR-like image. However, determining the appropriate fusion weights is difficult because each differently exposed image only contains a subset of the scene's details. When blending, the problem of local color inconsistency is more challenging; thus, it often requires manual tuning to avoid image artifacts. To address this problem, we present an adaptive coarse-to-fine searching approach to find the optimal fusion weights. In the coarse-tuning stage, fuzzy logic is used to efficiently decide the initial weights. In the fine-tuning stage, the multivariate normal conditional random field model is used to adjust the fuzzy-based initial weights which allows us to consider both intra- and inter-image information in the data. Moreover, a multiscale enhanced fusion scheme is proposed to blend input images when maintaining the details in each scale-level. The proposed fuzzy-based MNCRF (Multivariate Normal Conditional Random Fields) fusion method provided a smoother blending result and a more natural look. Meanwhile, the details in the highlighted and dark regions were preserved simultaneously. The experimental results demonstrated that our work outperformed the state-of-the-art methods not only in several objective quality measures but also in a user study analysis.