RFormer: Transformer-Based Generative Adversarial Network for Real Fundus Image Restoration on a New Clinical Benchmark

Deep learning segmentation of non-perfusion area from color fundus images and AI-generated fluorescein angiography

The non-perfusion area (NPA) of the retina is an important indicator in the visual prognosis of patients with branch retinal vein occlusion (BRVO). However, the current evaluation method of NPA, fluorescein angiography (FA), is invasive and burdensome. In this study, we examined the use of deep learning models for detecting NPA in color fundus images, bypassing the need for FA, and we also investigated the utility of synthetic FA generated from color fundus images. The models were evaluated using the Dice score and Monte Carlo dropout uncertainty. We retrospectively collected 403 sets of color fundus and FA images from 319 BRVO patients. We trained three deep learning models on FA, color fundus images, and synthetic FA. As a result, though the FA model achieved the highest score, the other two models also performed comparably. We found no statistical significance in median Dice scores between the models. However, the color fundus model showed significantly higher uncertainty than the other models (p < 0.05). In conclusion, deep learning models can detect NPAs from color fundus images with reasonable accuracy, though with somewhat less prediction stability. Synthetic FA stabilizes the prediction and reduces misleading uncertainty estimates by enhancing image quality.

FNeXter: A Multi-Scale Feature Fusion Network Based on ConvNeXt and Transformer for Retinal OCT Fluid Segmentation

The accurate segmentation and quantification of retinal fluid in Optical Coherence Tomography (OCT) images are crucial for the diagnosis and treatment of ophthalmic diseases such as age-related macular degeneration. However, the accurate segmentation of retinal fluid is challenging due to significant variations in the size, position, and shape of fluid, as well as their complex, curved boundaries. To address these challenges, we propose a novel multi-scale feature fusion attention network (FNeXter), based on ConvNeXt and Transformer, for OCT fluid segmentation. In FNeXter, we introduce a novel global multi-scale hybrid encoder module that integrates ConvNeXt, Transformer, and region-aware spatial attention. This module can capture long-range dependencies and non-local similarities while also focusing on local features. Moreover, this module possesses the spatial region-aware capabilities, enabling it to adaptively focus on the lesions regions. Additionally, we propose a novel self-adaptive multi-scale feature fusion attention module to enhance the skip connections between the encoder and the decoder. The inclusion of this module elevates the model's capacity to learn global features and multi-scale contextual information effectively. Finally, we conduct comprehensive experiments to evaluate the performance of the proposed FNeXter. Experimental results demonstrate that our proposed approach outperforms other state-of-the-art methods in the task of fluid segmentation.

Enhancing and Adapting in the Clinic: Source-Free Unsupervised Domain Adaptation for Medical Image Enhancement

Medical imaging provides many valuable clues involving anatomical structure and pathological characteristics. However, image degradation is a common issue in clinical practice, which can adversely impact the observation and diagnosis by physicians and algorithms. Although extensive enhancement models have been developed, these models require a well pre-training before deployment, while failing to take advantage of the potential value of inference data after deployment. In this paper, we raise an algorithm for source-free unsupervised domain adaptive medical image enhancement (SAME), which adapts and optimizes enhancement models using test data in the inference phase. A structure-preserving enhancement network is first constructed to learn a robust source model from synthesized training data. Then a teacher-student model is initialized with the source model and conducts source-free unsupervised domain adaptation (SFUDA) by knowledge distillation with the test data. Additionally, a pseudo-label picker is developed to boost the knowledge distillation of enhancement tasks. Experiments were implemented on ten datasets from three medical image modalities to validate the advantage of the proposed algorithm, and setting analysis and ablation studies were also carried out to interpret the effectiveness of SAME. The remarkable enhancement performance and benefits for downstream tasks demonstrate the potential and generalizability of SAME. The code is available at https://github.com/liamheng/Annotation-free-Medical-Image-Enhancement.

MAFE-Net: retinal vessel segmentation based on a multiple attention-guided fusion mechanism and ensemble learning network

The precise and automatic recognition of retinal vessels is of utmost importance in the prevention, diagnosis and assessment of certain eye diseases, yet it brings a nontrivial uncertainty for this challenging detection mission due to the presence of intricate factors, such as uneven and indistinct curvilinear shapes, unpredictable pathological deformations, and non-uniform contrast. Therefore, we propose a unique and practical approach based on a multiple attention-guided fusion mechanism and ensemble learning network (MAFE-Net) for retinal vessel segmentation. In conventional UNet-based models, long-distance dependencies are explicitly modeled, which may cause partial scene information loss. To compensate for the deficiency, various blood vessel features can be extracted from retinal images by using an attention-guided fusion module. In the skip connection part, a unique spatial attention module is applied to remove redundant and irrelevant information; this structure helps to better integrate low-level and high-level features. The final step involves a DropOut layer that removes some neurons randomly to prevent overfitting and improve generalization. Moreover, an ensemble learning framework is designed to detect retinal vessels by combining different deep learning models. To demonstrate the effectiveness of the proposed model, experimental results were verified in public datasets STARE, DRIVE, and CHASEDB1, which achieved F1 scores of 0.842, 0.825, and 0.814, and Accuracy values of 0.975, 0.969, and 0.975, respectively. Compared with eight state-of-the-art models, the designed model produces satisfactory results both visually and quantitatively.

Adaptive enhancement of cataractous retinal images for contrast standardization

Cataract affects the quality of fundus images, especially the contrast, due to lens opacity. In this paper, we propose a scheme to enhance different cataractous retinal images to the same contrast as normal images, which can automatically choose the suitable enhancement model based on cataract grading. A multi-level cataract dataset is constructed via the degradation model with quantified contrast. Then, an adaptive enhancement strategy is introduced to choose among three enhancement networks based on a blurriness classifier. The blurriness grading loss is proposed in the enhancement models to further constrain the contrast of the enhanced images. During test, the well-trained blurriness classifier can assist in the selection of enhancement networks with specific enhancement ability. Our method performs the best on the synthetic paired data on PSNR, SSIM, and FSIM and has the best PIQE and FID on 406 clinical fundus images. There is a 7.78% improvement for our method compared with the second on the introduced [Formula: see text] score without over-enhancement according to [Formula: see text], which demonstrates that the proper enhancement by our method is close to the high-quality images. The visual evaluation on multiple clinical datasets also shows the applicability of our method for different blurriness. The proposed method can benefit clinical diagnosis and improve the performance of computer-aided algorithms such as vessel tracking and vessel segmentation.

A generic fundus image enhancement network boosted by frequency self-supervised representation learning

Fundus photography is prone to suffer from image quality degradation that impacts clinical examination performed by ophthalmologists or intelligent systems. Though enhancement algorithms have been developed to promote fundus observation on degraded images, high data demands and limited applicability hinder their clinical deployment. To circumvent this bottleneck, a generic fundus image enhancement network (GFE-Net) is developed in this study to robustly correct unknown fundus images without supervised or extra data. Levering image frequency information, self-supervised representation learning is conducted to learn robust structure-aware representations from degraded images. Then with a seamless architecture that couples representation learning and image enhancement, GFE-Net can accurately correct fundus images and meanwhile preserve retinal structures. Comprehensive experiments are implemented to demonstrate the effectiveness and advantages of GFE-Net. Compared with state-of-the-art algorithms, GFE-Net achieves superior performance in data dependency, enhancement performance, deployment efficiency, and scale generalizability. Follow-up fundus image analysis is also facilitated by GFE-Net, whose modules are respectively verified to be effective for image enhancement.

Vision Transformers in Image Restoration: A Survey

The Vision Transformer (ViT) architecture has been remarkably successful in image restoration. For a while, Convolutional Neural Networks (CNN) predominated in most computer vision tasks. Now, both CNN and ViT are efficient approaches that demonstrate powerful capabilities to restore a better version of an image given in a low-quality format. In this study, the efficiency of ViT in image restoration is studied extensively. The ViT architectures are classified for every task of image restoration. Seven image restoration tasks are considered: Image Super-Resolution, Image Denoising, General Image Enhancement, JPEG Compression Artifact Reduction, Image Deblurring, Removing Adverse Weather Conditions, and Image Dehazing. The outcomes, the advantages, the limitations, and the possible areas for future research are detailed. Overall, it is noted that incorporating ViT in the new architectures for image restoration is becoming a rule. This is due to some advantages compared to CNN, such as better efficiency, especially when more data are fed to the network, robustness in feature extraction, and a better feature learning approach that sees better the variances and characteristics of the input. Nevertheless, some drawbacks exist, such as the need for more data to show the benefits of ViT over CNN, the increased computational cost due to the complexity of the self-attention block, a more challenging training process, and the lack of interpretability. These drawbacks represent the future research direction that should be targeted to increase the efficiency of ViT in the image restoration domain.