Green Fluorescent Protein and Phase-Contrast Image Fusion via Generative Adversarial Networks

MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer

Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

Generative Adversarial Networks in Medical Image Processing

BACKGROUND

The emergence of generative adversarial networks (GANs) has provided new technology and framework for the application of medical images. Specifically, a GAN requires little to no labeled data to obtain high-quality data that can be generated through competition between the generator and discriminator networks. Therefore, GANs are rapidly proving to be a state-of-the-art foundation, achieving enhanced performances in various medical applications.

METHODS

In this article, we introduce the principles of GANs and their various variants, deep convolutional GAN, conditional GAN, Wasserstein GAN, Info-GAN, boundary equilibrium GAN, and cycle-GAN.

RESULTS

All various GANs have found success in medical imaging tasks, including medical image enhancement, segmentation, classification, reconstruction, and synthesis. Furthermore, we summarize the data processing methods and evaluation indicators. Finally, we note the limitations of existing methods and the existing challenges that need to be addressed in this field.

CONCLUSION

Although GANs are in the initial stage of development in medical image processing, it will have a great prospect in the future.

A phase congruency-based green fluorescent protein and phase contrast image fusion method in nonsubsampled shearlet transform domain

Image fusion technique is an effective way to merge the information contained in different imaging modalities by generating a more informative composite image. Fusion of green fluorescent protein (GFP) and phase contrast images is of great significance to the subcellular localization, the functional analysis of protein, and the expression of gene. In this article, a phase congruency (PC)-based GFP and phase contrast image fusion method in nonsubsampled shearlet transform (NSST) domain is presented. The input images are decomposed by the NSST to acquire the multiscale and multidirection representations. The high-frequency coefficients are fused with a strategy based on PC and parameter-adaptive pulse coupled neural network (PA-PCNN), while the low-frequency coefficients are integrated through a local energy (LE)-based rule. Finally, the fused image is generated by conducting the inverse NSST on the merged high- and low-frequency coefficients. Experimental results illustrate that the presented method outperforms several state-of-the-art GFP and phase contrast image fusion algorithms on both qualitative and quantitative assessments.