Robust multimodal registration of fluorescein angiography and optical coherence tomography angiography images using evolutionary algorithms

FA4SANS-GAN: A Novel Machine Learning Generative Adversarial Network to Further Understand Ophthalmic Changes in Spaceflight Associated Neuro-Ocular Syndrome (SANS)

Purpose

To provide an automated system for synthesizing fluorescein angiography (FA) images from color fundus photographs for averting risks associated with fluorescein dye and extend its future application to spaceflight associated neuro-ocular syndrome (SANS) detection in spaceflight where resources are limited.

Design

Development and validation of a novel conditional generative adversarial network (GAN) trained on limited amount of FA and color fundus images with diabetic retinopathy and control cases.

Participants

Color fundus and FA paired images for unique patients were collected from a publicly available study.

Methods

FA4SANS-GAN was trained to generate FA images from color fundus photographs using 2 multiscale generators coupled with 2 patch-GAN discriminators. Eight hundred fifty color fundus and FA images were utilized for training by augmenting images from 17 unique patients. The model was evaluated on 56 fluorescein images collected from 14 unique patients. In addition, it was compared with 3 other GAN architectures trained on the same data set. Furthermore, we test the robustness of the models against acquisition noise and retaining structural information when introduced to artificially created biological markers.

Main Outcome Measures

For GAN synthesis, metric Fréchet Inception Distance (FID) and Kernel Inception Distance (KID). Also, two 1-sided tests (TOST) based on Welch's t test for measuring statistical significance.

Results

On test FA images, mean FID for FA4SANS-GAN was 39.8 (standard deviation, 9.9), which is better than GANgio model's mean of 43.2 (standard deviation, 13.7), Pix2PixHD's mean of 57.3 (standard deviation, 11.5) and Pix2Pix's mean of 67.5 (standard deviation, 11.7). Similarly for KID, FA4SANS-GAN achieved mean of 0.00278 (standard deviation, 0.00167) which is better than other 3 model's mean KID of 0.00303 (standard deviation, 0.00216), 0.00609 (standard deviation, 0.00238), 0.00784 (standard deviation, 0.00218). For TOST measurement, FA4SANS-GAN was proven to be statistically significant versus GANgio (P = 0.006); versus Pix2PixHD (P < 0.00001); and versus Pix2Pix (P < 0.00001).

Conclusions

Our study has shown FA4SANS-GAN to be statistically significant for 2 GAN synthesis metrics. Moreover, it is robust against acquisition noise, and can retain clear biological markers compared with the other 3 GAN architectures. This deployment of this model can be crucial in the International Space Station for detecting SANS.

Financial Disclosures

The authors have no proprietary or commercial interest in any materials discussed in this article.

BCSwinReg: A cross-modal attention network for CBCT-to-CT multimodal image registration

Computed tomography (CT) and cone beam computed tomography (CBCT) registration plays an important role in radiotherapy. However, the poor quality of CBCT makes CBCT-CT multimodal registration challenging. Effective feature fusion and mapping often lead to better registration results for multimodal registration. Therefore, we proposed a new backbone network BCSwinReg and a cross-modal attention module CrossSwin. Specifically, a cross-modal attention CrossSwin is designed to promote multi-modal feature fusion, map the multi-modal domain to the common domain, and thus helping the network learn the correspondence between images better. Furthermore, a new network, BCSwinReg, is proposed to discover correspondence through cross-attention exchange information, obtain multi-level semantic information through a multi-resolution strategy, and finally integrate the deformation of multi-resolutions by the divide-conquer cascade method. We performed experiments on the publicly available 4D-Lung dataset to demonstrate the effectiveness of CrossSwin and BCSwinReg. Compared with VoxelMorph, the BCSwinReg has obtained performance improvements of 3.3% in Dice Similarity Coefficient (DSC) and 0.19 in the average 95% Hausdorff distance (HD95).

White shark optimizer with optimal deep learning based effective unmanned aerial vehicles communication and scene classification

Unmanned aerial vehicles (UAVs) become a promising enabler for the next generation of wireless networks with the tremendous growth in electronics and communications. The application of UAV communications comprises messages relying on coverage extension for transmission networks after disasters, Internet of Things (IoT) devices, and dispatching distress messages from the device positioned within the coverage hole to the emergency centre. But there are some problems in enhancing UAV clustering and scene classification using deep learning approaches for enhancing performance. This article presents a new White Shark Optimizer with Optimal Deep Learning based Effective Unmanned Aerial Vehicles Communication and Scene Classification (WSOODL-UAVCSC) technique. UAV clustering and scene categorization present many deep learning challenges in disaster management: scene understanding complexity, data variability and abundance, visual data feature extraction, nonlinear and high-dimensional data, adaptability and generalization, real-time decision making, UAV clustering optimization, sparse and incomplete data. the need to handle complex, high-dimensional data, adapt to changing environments, and make quick, correct decisions in critical situations drives deep learning in UAV clustering and scene categorization. The purpose of the WSOODL-UAVCSC technique is to cluster the UAVs for effective communication and scene classification. The WSO algorithm is utilized for the optimization of the UAV clustering process and enables to accomplish effective communication and interaction in the network. With dynamic adjustment of the clustering, the WSO algorithm improves the performance and robustness of the UAV system. For the scene classification process, the WSOODL-UAVCSC technique involves capsule network (CapsNet) feature extraction, marine predators algorithm (MPA) based hyperparameter tuning, and echo state network (ESN) classification. A wide-ranging simulation analysis was conducted to validate the enriched performance of the WSOODL-UAVCSC approach. Extensive result analysis pointed out the enhanced performance of the WSOODL-UAVCSC method over other existing techniques. The WSOODL-UAVCSC method achieved an accuracy of 99.12%, precision of 97.45%, recall of 98.90%, and F1-score of 98.10% when compared to other existing techniques.

Assessment of a Novel Semi-Automated Algorithm for the Quantification of the Parafoveal Capillary Network

Introduction

We present a novel semi-automated computerized method for the detection and quantification of parafoveal capillary network (PCN) in fluorescein angiography (FA) images.

Material and Methods

An algorithm detecting the superficial parafoveal capillary bed in high-resolution grayscale FA images and creating a one-pixel-wide PCN skeleton was developed using MatLab software. In addition to PCN detection, capillary density and branch point density in two circular areas centered on the center of the foveal avascular zone of 500μm and 750μm radius was calculated by the algorithm. Three consecutive FA images with distinguishable PCN from 56 eyes from 56 subjects were used for analysis. Both manual and semi-automated detection of the PCN and branch points was performed and compared. Three different intensity thresholds were used for the PCN detection to optimize the method defined as mean(I)+0.05*SD(I), mean(I) and mean(I)-0.05*SD(I), where I is the grayscale intensity of each image and SD the standard deviation. Limits of agreement (LoA), intraclass correlation coefficient (ICC) and Pearson's correlation coefficient (r) were calculated.

Results

Using mean(I)-0.05*SD(I) as threshold the average difference in PCN density between semi-automated and manual method was 0.197 (0.316) deg^-1 at 500μm radius and 0.409 (0.562) deg^-1 at 750μm radius. The LoA were -0.421 to 0.817 and -0.693 to 1.510 deg^-1, respectively. The average difference of branch point density between semi-automated and manual method was zero for both areas; LoA were -0.001 to 0.002 and -0.001 to 0.001 branch points/degrees², respectively. The other two intensity thresholds provided wider LoA for both metrics. The semi-automated algorithm showed great repeatability (ICC>0.91 in the 500μm radius and ICC>0.84 in the 750μm radius) for both metrics.

Conclusion

This semi-automated algorithm seems to provide readings in agreement with those of manual capillary tracing in FA. Larger prospective studies are needed to confirm the utility of the algorithm in clinical practice.

Deformable registration of multimodal retinal images using a weakly supervised deep learning approach

There are different retinal vascular imaging modalities widely used in clinical practice to diagnose different retinal pathologies. The joint analysis of these multimodal images is of increasing interest since each of them provides common and complementary visual information. However, if we want to facilitate the comparison of two images, obtained with different techniques and containing the same retinal region of interest, it will be necessary to make a previous registration of both images. Here, we present a weakly supervised deep learning methodology for robust deformable registration of multimodal retinal images, which is applied to implement a method for the registration of fluorescein angiography (FA) and optical coherence tomography angiography (OCTA) images. This methodology is strongly inspired by VoxelMorph, a general unsupervised deep learning framework of the state of the art for deformable registration of unimodal medical images. The method was evaluated in a public dataset with 172 pairs of FA and superficial plexus OCTA images. The degree of alignment of the common information (blood vessels) and preservation of the non-common information (image background) in the transformed image were measured using the Dice coefficient (DC) and zero-normalized cross-correlation (ZNCC), respectively. The average values of the mentioned metrics, including the standard deviations, were DC = 0.72  ±  0.10 and ZNCC = 0.82  ±  0.04. The time required to obtain each pair of registered images was 0.12 s. These results outperform rigid and deformable registration methods with which our method was compared.

Evolutionary Image Registration: A Review

Image registration is one of the most important image processing tools enabling recognition, classification, detection and other analysis tasks. Registration methods are used to solve a large variety of real-world problems, including remote sensing, computer vision, geophysics, medical image analysis, surveillance, and so on. In the last few years, nature-inspired algorithms and metaheuristics have been successfully used to address the image registration problem, becoming a solid alternative for direct optimization methods. The aim of this paper is to investigate and summarize a series of state-of-the-art works reporting evolutionary-based registration methods. The papers were selected using the PRISMA 2020 method. The reported algorithms are reviewed and compared in terms of evolutionary components, fitness function, image similarity measures and algorithm accuracy indexes used in the alignment process.