Leveraging Regular Fundus Images for Training UWF Fundus Diagnosis Models via Adversarial Learning and Pseudo-Labeling

Cross-modality transfer learning with knowledge infusion for diabetic retinopathy grading

Background

Ultra-wide-field (UWF) fundus photography represents an emerging retinal imaging technique offering a broader field of view, thus enhancing its utility in screening and diagnosing various eye diseases, notably diabetic retinopathy (DR). However, the application of computer-aided diagnosis for DR using UWF images confronts two major challenges. The first challenge arises from the limited availability of labeled UWF data, making it daunting to train diagnostic models due to the high cost associated with manual annotation of medical images. Secondly, existing models' performance requires enhancement due to the absence of prior knowledge to guide the learning process.

Purpose

By leveraging extensively annotated datasets within the field, which encompass large-scale, high-quality color fundus image datasets annotated at either image-level or pixel-level, our objective is to transfer knowledge from these datasets to our target domain through unsupervised domain adaptation.

Methods

Our approach presents a robust model for assessing the severity of diabetic retinopathy (DR) by leveraging unsupervised lesion-aware domain adaptation in ultra-wide-field (UWF) images. Furthermore, to harness the wealth of detailed annotations in publicly available color fundus image datasets, we integrate an adversarial lesion map generator. This generator supplements the grading model by incorporating auxiliary lesion information, drawing inspiration from the clinical methodology of evaluating DR severity by identifying and quantifying associated lesions.

Results

We conducted both quantitative and qualitative evaluations of our proposed method. In particular, among the six representative DR grading methods, our approach achieved an accuracy (ACC) of 68.18% and a precision (pre) of 67.43%. Additionally, we conducted extensive experiments in ablation studies to validate the effectiveness of each component of our proposed method.

Conclusion

In conclusion, our method not only improves the accuracy of DR grading, but also enhances the interpretability of the results, providing clinicians with a reliable DR grading scheme.

Generative models improve fairness of medical classifiers under distribution shifts

Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and 'labeling' by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.

A new segmentation algorithm for peripapillary atrophy and optic disk from ultra-widefield Photographs1

BACKGROUND AND OBJECTIVE

The prevalence of myopia and high myopia is increasing globally, underscoring the growing importance of diagnosing high myopia-related pathologies. While existing image segmentation models, such as U-Net, UNet++, ResU-Net, and TransUNet, have achieved significant success in medical image segmentation, they still face challenges when dealing with ultra-widefield (UWF) fundus images. This study introduces a novel automatic segmentation algorithm for the optic disc and peripapillary atrophy (PPA) based on UWF fundus images, aimed at assisting ophthalmologists in more accurately diagnosing high myopia-related diseases.

METHODS

In this study, we developed a segmentation model leveraging a Transformer-based network structure, complemented by atrous convolution and selective boundary aggregation modules, to elevate the accuracy of segmenting the optic disc and PPA in UWF photography. The atrous convolution module adeptly manages multi-scale features, catering to the variances in target sizes and expanding the deep network's receptive field. Concurrently, the incorporation of the selective boundary aggregation module in the skip connections of the model significantly improves the differentiation of boundary information between segmentation targets. Moreover, the comparison of our proposed algorithm with classical segmentation models like U-Net, UNet++, ResU-Net, and TransUNet highlights its considerable advantages in processing UWF photographs.

RESULTS

The experimental results show that, compared to the other four models, our algorithm demonstrates substantial improvements in segmenting the optic disc and PPA in UWF photographs. In PPA segmentation, our algorithm improves by 0.8% in Dice, 1.8% in sensitivity, and 1.3% in intersection over union (IOU). In optic disc segmentation, our algorithm improves by 0.3% in Dice, 0.6% in precision, and 0.4% in IOU.

CONCLUSION

Our proposed method improves the segmentation accuracy of PPA and optic disks based on UWF photographs, which is valuable for diagnosing high myopia-related diseases in ophthalmology clinics.

Addressing inter-device variations in optical coherence tomography angiography: will image-to-image translation systems help?

BACKGROUND

Optical coherence tomography angiography (OCTA) is an innovative technology providing visual and quantitative data on retinal microvasculature in a non-invasive manner.

MAIN BODY

Due to variations in the technical specifications of different OCTA devices, there are significant inter-device differences in OCTA data, which can limit their comparability and generalizability. These variations can also result in a domain shift problem that may interfere with applicability of machine learning models on data obtained from different OCTA machines. One possible approach to address this issue may be unsupervised deep image-to-image translation leveraging systems such as Cycle-Consistent Generative Adversarial Networks (Cycle-GANs) and Denoising Diffusion Probabilistic Models (DDPMs). Through training on unpaired images from different device domains, Cycle-GANs and DDPMs may enable cross-domain translation of images. They have been successfully applied in various medical imaging tasks, including segmentation, denoising, and cross-modality image-to-image translation. In this commentary, we briefly describe how Cycle-GANs and DDPMs operate, and review the recent experiments with these models on medical and ocular imaging data. We then discuss the benefits of applying such techniques for inter-device translation of OCTA data and the potential challenges ahead.

CONCLUSION

Retinal imaging technologies and deep learning-based domain adaptation techniques are rapidly evolving. We suggest exploring the potential of image-to-image translation methods in improving the comparability of OCTA data from different centers or devices. This may facilitate more efficient analysis of heterogeneous data and broader applicability of machine learning models trained on limited datasets in this field.

Unpaired, unsupervised domain adaptation assumes your domains are already similar

Unsupervised domain adaptation is a popular method in medical image analysis, but it can be tricky to make it work: without labels to link the domains, domains must be matched using feature distributions. If there is no additional information, this often leaves a choice between multiple possibilities to map the data that may be equally likely but not equally correct. In this paper we explore the fundamental problems that may arise in unsupervised domain adaptation, and discuss conditions that might still make it work. Focusing on medical image analysis, we argue that images from different domains may have similar class balance, similar intensities, similar spatial structure, or similar textures. We demonstrate how these implicit conditions can affect domain adaptation performance in experiments with synthetic data, MNIST digits, and medical images. We observe that practical success of unsupervised domain adaptation relies on existing similarities in the data, and is anything but guaranteed in the general case. Understanding these implicit assumptions is a key step in identifying potential problems in domain adaptation and improving the reliability of the results.

Deep Active Learning for Automatic Mitotic Cell Detection on HEp-2 Specimen Medical Images

Identifying Human Epithelial Type 2 (HEp-2) mitotic cells is a crucial procedure in anti-nuclear antibodies (ANAs) testing, which is the standard protocol for detecting connective tissue diseases (CTD). Due to the low throughput and labor-subjectivity of the ANAs' manual screening test, there is a need to develop a reliable HEp-2 computer-aided diagnosis (CAD) system. The automatic detection of mitotic cells from the microscopic HEp-2 specimen images is an essential step to support the diagnosis process and enhance the throughput of this test. This work proposes a deep active learning (DAL) approach to overcoming the cell labeling challenge. Moreover, deep learning detectors are tailored to automatically identify the mitotic cells directly in the entire microscopic HEp-2 specimen images, avoiding the segmentation step. The proposed framework is validated using the I3A Task-2 dataset over 5-fold cross-validation trials. Using the YOLO predictor, promising mitotic cell prediction results are achieved with an average of 90.011% recall, 88.307% precision, and 81.531% mAP. Whereas, average scores of 86.986% recall, 85.282% precision, and 78.506% mAP are obtained using the Faster R-CNN predictor. Employing the DAL method over four labeling rounds effectively enhances the accuracy of the data annotation, and hence, improves the prediction performance. The proposed framework could be practically applicable to support medical personnel in making rapid and accurate decisions about the mitotic cells' existence.

Semantically preserving adversarial unsupervised domain adaptation network for improving disease recognition from chest x-rays

Supervised deep learning methods have been successfully applied in medical imaging. However, training deep learning systems often requires ample annotated data. Due to cost and time restrictions, not all collected medical images, e.g., chest x-rays (CXRs), can be labeled in practice. To classify these unlabeled images, a solution may involve adopting a model trained with sufficient labeled data in relevant domains (with both source and target being CXRs). However, domain shift may cause the trained model not able to generalize well on unlabeled target datasets. This work aims to develop a novel unsupervised domain adaptation (UDA) framework to improve recognition performance on unlabeled target data. We present a semantically preserving adversarial UDA network, i.e., SPA-UDA net, with the potential to bridge the domain gap, by reconstructing the images in the target domain via an adversarial encode-and-reconstruct translation architecture. To preserve the class-specific semantic information (i.e., with or without disease) of the original images when translating, a semantically consistent framework is embedded. This framework is designed to guarantee that fine-grained disease-related information on the original images can be safely transferred. Furthermore, the proposed SPA-UDA net does not require paired images from source and target domains when training, which reduces the cost of arranging data significantly and is ideal for UDA. We evaluate the proposed SPA-UDA net on two public CXR datasets for lung disease recognition. The experimental results show that the proposed framework achieves significant performance improvements compared to other state-of-the-art UDA methods.

Medical image data augmentation: techniques, comparisons and interpretations

Designing deep learning based methods with medical images has always been an attractive area of research to assist clinicians in rapid examination and accurate diagnosis. Those methods need a large number of datasets including all variations in their training stages. On the other hand, medical images are always scarce due to several reasons, such as not enough patients for some diseases, patients do not want to allow their images to be used, lack of medical equipment or equipment, inability to obtain images that meet the desired criteria. This issue leads to bias in datasets, overfitting, and inaccurate results. Data augmentation is a common solution to overcome this issue and various augmentation techniques have been applied to different types of images in the literature. However, it is not clear which data augmentation technique provides more efficient results for which image type since different diseases are handled, different network architectures are used, and these architectures are trained and tested with different numbers of data sets in the literature. Therefore, in this work, the augmentation techniques used to improve performances of deep learning based diagnosis of the diseases in different organs (brain, lung, breast, and eye) from different imaging modalities (MR, CT, mammography, and fundoscopy) have been examined. Also, the most commonly used augmentation methods have been implemented, and their effectiveness in classifications with a deep network has been discussed based on quantitative performance evaluations. Experiments indicated that augmentation techniques should be chosen carefully according to image types.

A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning

This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.

Deep learning and sub-band fluorescence imaging-based method for caries and calculus diagnosis embeddable on different smartphones

Popularizing community and home early caries screening is essential for caries prevention and treatment. However, a high-precision, low-cost, and portable automated screening tool is currently lacking. This study constructed an automated diagnosis model for dental caries and calculus using fluorescence sub-band imaging combined with deep learning. The proposed method is divided into two stages: the first stage collects imaging information of dental caries in different fluorescence spectral bands and obtains six-channel fluorescence images. The second stage employs a 2-D-3-D hybrid convolutional neural network combined with the attention mechanism for classification and diagnosis. The experiments demonstrate that the method has competitive performance compared to existing methods. In addition, the feasibility of transferring this approach to different smartphones is discussed. This highly accurate, low-cost, portable method has potential applications in community and at-home caries detection.

LAC-GAN: Lesion attention conditional GAN for Ultra-widefield image synthesis

Automatic detection of retinal diseases based on deep learning technology and Ultra-widefield (UWF) images plays an important role in clinical practices in recent years. However, due to small lesions and limited data samples, it is not easy to train a detection-accurate model with strong generalization ability. In this paper, we propose a lesion attention conditional generative adversarial network (LAC-GAN) to synthesize retinal images with realistic lesion details to improve the training of the disease detection model. Specifically, the generator takes the vessel mask and class label as the conditional inputs, and processes the random Gaussian noise by a series of residual block to generate the synthetic images. To focus on pathological information, we propose a lesion feature attention mechanism based on random forest (RF) method, which constructs its reverse activation network to activate the lesion features. For discriminator, a weight-sharing multi-discriminator is designed to improve the performance of model by affine transformations. Experimental results on multi-center UWF image datasets demonstrate that the proposed method can generate retinal images with reasonable details, which helps to enhance the performance of the disease detection model.

Deep learning for ultra-widefield imaging: a scoping review

PURPOSE

This article is a scoping review of published and peer-reviewed articles using deep-learning (DL) applied to ultra-widefield (UWF) imaging. This study provides an overview of the published uses of DL and UWF imaging for the detection of ophthalmic and systemic diseases, generative image synthesis, quality assessment of images, and segmentation and localization of ophthalmic image features.

METHODS

A literature search was performed up to August 31st, 2021 using PubMed, Embase, Cochrane Library, and Google Scholar. The inclusion criteria were as follows: (1) deep learning, (2) ultra-widefield imaging. The exclusion criteria were as follows: (1) articles published in any language other than English, (2) articles not peer-reviewed (usually preprints), (3) no full-text availability, (4) articles using machine learning algorithms other than deep learning. No study design was excluded from consideration.

RESULTS

A total of 36 studies were included. Twenty-three studies discussed ophthalmic disease detection and classification, 5 discussed segmentation and localization of ultra-widefield images (UWFIs), 3 discussed generative image synthesis, 3 discussed ophthalmic image quality assessment, and 2 discussed detecting systemic diseases via UWF imaging.

CONCLUSION

The application of DL to UWF imaging has demonstrated significant effectiveness in the diagnosis and detection of ophthalmic diseases including diabetic retinopathy, retinal detachment, and glaucoma. DL has also been applied in the generation of synthetic ophthalmic images. This scoping review highlights and discusses the current uses of DL with UWF imaging, and the future of DL applications in this field.

Improving foveal avascular zone segmentation in fluorescein angiograms by leveraging manual vessel labels from public color fundus pictures

In clinical routine, ophthalmologists frequently analyze the shape and size of the foveal avascular zone (FAZ) to detect and monitor retinal diseases. In order to extract those parameters, the contours of the FAZ need to be segmented, which is normally achieved by analyzing the retinal vasculature (RV) around the macula in fluorescein angiograms (FA). Computer-aided segmentation methods based on deep learning (DL) can automate this task. However, current approaches for segmenting the FAZ are often tailored to a specific dataset or require manual initialization. Furthermore, they do not take the variability and challenges of clinical FA into account, which are often of low quality and difficult to analyze. In this paper we propose a DL-based framework to automatically segment the FAZ in challenging FA scans from clinical routine. Our approach mimics the workflow of retinal experts by using additional RV labels as a guidance during training. Hence, our model is able to produce RV segmentations simultaneously. We minimize the annotation work by using a multi-modal approach that leverages already available public datasets of color fundus pictures (CFPs) and their respective manual RV labels. Our experimental evaluation on two datasets with FA from 1) clinical routine and 2) large multicenter clinical trials shows that the addition of weak RV labels as a guidance during training improves the FAZ segmentation significantly with respect to using only manual FAZ annotations.

Generative Adversarial Networks in Medical Image augmentation: A review

OBJECT

With the development of deep learning, the number of training samples for medical image-based diagnosis and treatment models is increasing. Generative Adversarial Networks (GANs) have attracted attention in medical image processing due to their excellent image generation capabilities and have been widely used in data augmentation. In this paper, a comprehensive and systematic review and analysis of medical image augmentation work are carried out, and its research status and development prospects are reviewed.

METHOD

This paper reviews 105 medical image augmentation related papers, which mainly collected by ELSEVIER, IEEE Xplore, and Springer from 2018 to 2021. We counted these papers according to the parts of the organs corresponding to the images, and sorted out the medical image datasets that appeared in them, the loss function in model training, and the quantitative evaluation metrics of image augmentation. At the same time, we briefly introduce the literature collected in three journals and three conferences that have received attention in medical image processing.

RESULT

First, we summarize the advantages of various augmentation models, loss functions, and evaluation metrics. Researchers can use this information as a reference when designing augmentation tasks. Second, we explore the relationship between augmented models and the amount of the training set, and tease out the role that augmented models may play when the quality of the training set is limited. Third, the statistical number of papers shows that the development momentum of this research field remains strong. Furthermore, we discuss the existing limitations of this type of model and suggest possible research directions.

CONCLUSION

We discuss GAN-based medical image augmentation work in detail. This method effectively alleviates the challenge of limited training samples for medical image diagnosis and treatment models. It is hoped that this review will benefit researchers interested in this field.

Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey

BACKGROUND

Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions.

METHODS

We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN.

RESULTS

In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns.

CONCLUSIONS

The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.

Unsupervised Domain Adaptation Based Image Synthesis and Feature Alignment for Joint Optic Disc and Cup Segmentation

Due to the discrepancy of different devices for fundus image collection, a well-trained neural network is usually unsuitable for another new dataset. To solve this problem, the unsupervised domain adaptation strategy attracts a lot of attentions. In this paper, we propose an unsupervised domain adaptation method based image synthesis and feature alignment (ISFA) method to segment optic disc and cup on the fundus image. The GAN-based image synthesis (IS) mechanism along with the boundary information of optic disc and cup is utilized to generate target-like query images, which serves as the intermediate latent space between source domain and target domain images to alleviate the domain shift problem. Specifically, we use content and style feature alignment (CSFA) to ensure the feature consistency among source domain images, target-like query images and target domain images. The adversarial learning is used to extract domain invariant features for output-level feature alignment (OLFA). To enhance the representation ability of domain-invariant boundary structure information, we introduce the edge attention module (EAM) for low-level feature maps. Eventually, we train our proposed method on the training set of the REFUGE challenge dataset and test it on Drishti-GS and RIM-ONE_r3 datasets. On the Drishti-GS dataset, our method achieves about 3% improvement of Dice on optic cup segmentation over the next best method. We comprehensively discuss the robustness of our method for small dataset domain adaptation. The experimental results also demonstrate the effectiveness of our method. Our code is available at https://github.com/thinkobj/ISFA.