Retinal Fundus Multi-Disease Image Dataset (RFMiD): A Dataset for Multi-Disease Detection Research

Efficient pyramid channel attention network for pathological myopia recognition with pretraining-and-finetuning

Pathological myopia (PM) is the leading ocular disease for impaired vision worldwide. Clinically, the characteristics of pathology distribution in PM are global-local on the fundus image, which plays a significant role in assisting clinicians in diagnosing PM. However, most existing deep neural networks focused on designing complex architectures but rarely explored the pathology distribution prior of PM. To tackle this issue, we propose an efficient pyramid channel attention (EPCA) module, which fully leverages the potential of the clinical pathology prior of PM with pyramid pooling and multi-scale context fusion. Then, we construct EPCA-Net for automatic PM recognition based on fundus images by stacking a sequence of EPCA modules. Moreover, motivated by the recent pretraining-and-finetuning paradigm, we attempt to adapt pre-trained natural image models for PM recognition by freezing them and treating the EPCA and other attention modules as adapters. In addition, we construct a PM recognition benchmark termed PM-fundus by collecting fundus images of PM from publicly available datasets. The comprehensive experiments demonstrate the superiority of EPCA-Net over state-of-the-art methods in the PM recognition task. For example, EPCA-Net achieves 97.56% accuracy and outperforms ViT by 2.85% accuracy on the PM-fundus dataset. The results also show that our method based on the pretraining-and-finetuning paradigm achieves competitive performance through comparisons to part of previous methods based on traditional fine-tuning paradigm with fewer tunable parameters, which has the potential to leverage more natural image foundation models to address the PM recognition task in limited medical data regime.

A deep learning framework for the early detection of multi-retinal diseases

Retinal images play a pivotal contribution to the diagnosis of various ocular conditions by ophthalmologists. Extensive research was conducted to enable early detection and timely treatment using deep learning algorithms for retinal fundus images. Quick diagnosis and treatment planning can be facilitated by deep learning models' ability to process images rapidly and deliver outcomes instantly. Our research aims to provide a non-invasive method for early detection and timely eye disease treatment using a Convolutional Neural Network (CNN). We used a dataset Retinal Fundus Multi-disease Image Dataset (RFMiD), which contains various categories of fundus images representing different eye diseases, including Media Haze (MH), Optic Disc Cupping (ODC), Diabetic Retinopathy (DR), and healthy images (WNL). Several pre-processing techniques were applied to improve the model's performance, such as data augmentation, cropping, resizing, dataset splitting, converting images to arrays, and one-hot encoding. CNNs have extracted extract pertinent features from the input color fundus images. These extracted features are employed to make predictive diagnostic decisions. In this article three CNN models were used to perform experiments. The model's performance is assessed utilizing statistical metrics such as accuracy, F1 score, recall, and precision. Based on the results, the developed framework demonstrates promising performance with accuracy rates of up to 89.81% for validation and 88.72% for testing using 12-layer CNN after Data Augmentation. The accuracy rate obtained from 20-layer CNN is 90.34% for validation and 89.59% for testing with Augmented data. The accuracy obtained from 20-layer CNN is greater but this model shows overfitting. These accuracy rates suggested that the deep learning model has learned to distinguish between different eye disease categories and healthy images effectively. This study's contribution lies in providing a reliable and efficient diagnostic system for the simultaneous detection of multiple eye diseases through the analysis of color fundus images.

Multi-modality multi-label ocular abnormalities detection with transformer-based semantic dictionary learning

Blindness is preventable by early detection of ocular abnormalities. Computer-aided diagnosis for ocular abnormalities is built by analyzing retinal imaging modalities, for instance, Color Fundus Photography (CFP). This research aims to propose a multi-label detection of 28 ocular abnormalities consisting of frequent and rare abnormalities from a single CFP by using transformer-based semantic dictionary learning. Rare labels are usually ignored because of a lack of features. We tackle this condition by adding the co-occurrence dependency factor to the model from the linguistic features of the labels. The model learns the relation between spatial features and linguistic features represented as a semantic dictionary. The proposed method treats the semantic dictionary as one of the main important parts of the model. It acts as the query while the spatial features are the key and value. The experiments are conducted on the RFMiD dataset. The results show that the proposed method achieves the top 30% in Evaluation Set on the RFMiD dataset challenge. It also shows that treating the semantic dictionary as one of the strong factors in model detection increases the performance when compared with the method that treats the semantic dictionary as a weak factor.

Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism

The transformative role of artificial intelligence (AI) in various fields highlights the need for it to be both accurate and fair. Biased medical AI systems pose significant potential risks to achieving fair and equitable healthcare. Here, we show an implicit fairness learning approach to build a fairer ophthalmology AI (called FairerOPTH) that mitigates sex (biological attribute) and age biases in AI diagnosis of eye diseases. Specifically, FairerOPTH incorporates the causal relationship between fundus features and eye diseases, which is relatively independent of sensitive attributes such as race, sex, and age. We demonstrate on a large and diverse collected dataset that FairerOPTH significantly outperforms several state-of-the-art approaches in terms of diagnostic accuracy and fairness for 38 eye diseases in ultra-widefield imaging and 16 eye diseases in narrow-angle imaging. This work demonstrates the significant potential of implicit fairness learning in promoting equitable treatment for patients regardless of their sex or age.

PallorMetrics: Software for Automatically Quantifying Optic Disc Pallor in Fundus Photographs, and Associations With Peripapillary RNFL Thickness

Purpose

We sough to develop an automatic method of quantifying optic disc pallor in fundus photographs and determine associations with peripapillary retinal nerve fiber layer (pRNFL) thickness.

Methods

We used deep learning to segment the optic disc, fovea, and vessels in fundus photographs, and measured pallor. We assessed the relationship between pallor and pRNFL thickness derived from optical coherence tomography scans in 118 participants. Separately, we used images diagnosed by clinical inspection as pale (n = 45) and assessed how measurements compared with healthy controls (n = 46). We also developed automatic rejection thresholds and tested the software for robustness to camera type, image format, and resolution.

Results

We developed software that automatically quantified disc pallor across several zones in fundus photographs. Pallor was associated with pRNFL thickness globally (β = -9.81; standard error [SE] = 3.16; P < 0.05), in the temporal inferior zone (β = -29.78; SE = 8.32; P < 0.01), with the nasal/temporal ratio (β = 0.88; SE = 0.34; P < 0.05), and in the whole disc (β = -8.22; SE = 2.92; P < 0.05). Furthermore, pallor was significantly higher in the patient group. Last, we demonstrate the analysis to be robust to camera type, image format, and resolution.

Conclusions

We developed software that automatically locates and quantifies disc pallor in fundus photographs and found associations between pallor measurements and pRNFL thickness.

Translational Relevance

We think our method will be useful for the identification, monitoring, and progression of diseases characterized by disc pallor and optic atrophy, including glaucoma, compression, and potentially in neurodegenerative disorders.

Smart grading of diabetic retinopathy: an intelligent recommendation-based fine-tuned EfficientNetB0 framework

Diabetic retinopathy is a condition that affects the retina and causes vision loss due to blood vessel destruction. The retina is the layer of the eye responsible for visual processing and nerve signaling. Diabetic retinopathy causes vision loss, floaters, and sometimes blindness; however, it often shows no warning signals in the early stages. Deep learning-based techniques have emerged as viable options for automated illness classification as large-scale medical imaging datasets have become more widely available. To adapt to medical image analysis tasks, transfer learning makes use of pre-trained models to extract high-level characteristics from natural images. In this research, an intelligent recommendation-based fine-tuned EfficientNetB0 model has been proposed for quick and precise assessment for the diagnosis of diabetic retinopathy from fundus images, which will help ophthalmologists in early diagnosis and detection. The proposed EfficientNetB0 model is compared with three transfer learning-based models, namely, ResNet152, VGG16, and DenseNet169. The experimental work is carried out using publicly available datasets from Kaggle consisting of 3,200 fundus images. Out of all the transfer learning models, the EfficientNetB0 model has outperformed with an accuracy of 0.91, followed by DenseNet169 with an accuracy of 0.90. In comparison to other approaches, the proposed intelligent recommendation-based fine-tuned EfficientNetB0 approach delivers state-of-the-art performance on the accuracy, recall, precision, and F1-score criteria. The system aims to assist ophthalmologists in early detection, potentially alleviating the burden on healthcare units.

Visual acuity prediction on real-life patient data using a machine learning based multistage system

In ophthalmology, intravitreal operative medication therapy (IVOM) is a widespread treatment for diseases related to the age-related macular degeneration (AMD), the diabetic macular edema, as well as the retinal vein occlusion. However, in real-world settings, patients often suffer from loss of vision on time scales of years despite therapy, whereas the prediction of the visual acuity (VA) and the earliest possible detection of deterioration under real-life conditions is challenging due to heterogeneous and incomplete data. In this contribution, we present a workflow for the development of a research-compatible data corpus fusing different IT systems of the department of ophthalmology of a German maximum care hospital. The extensive data corpus allows predictive statements of the expected progression of a patient and his or her VA in each of the three diseases. For the disease AMD, we found out a significant deterioration of the visual acuity over time. Within our proposed multistage system, we subsequently classify the VA progression into the three groups of therapy "winners", "stabilizers", and "losers" (WSL classification scheme). Our OCT biomarker classification using an ensemble of deep neural networks results in a classification accuracy (F1-score) of over 98%, enabling us to complete incomplete OCT documentations while allowing us to exploit them for a more precise VA modelling process. Our VA prediction requires at least four VA examinations and optionally OCT biomarkers from the same time period to predict the VA progression within a forecasted time frame, whereas our prediction is currently restricted to IVOM/no therapy. We achieve a final prediction accuracy of 69% in macro average F1-score, while being in the same range as the ophthalmologists with 57.8 and 50 ± 10.7 % F1-score.

Improved predictive diagnosis of diabetic macular edema based on hybrid models: An observational study

Diabetic Macular Edema (DME) is the most common sight-threatening complication of type 2 diabetes. Optical Coherence Tomography (OCT) is the most useful imaging technique to diagnose, follow up, and evaluate treatments for DME. However, OCT exam and devices are expensive and unavailable in all clinics in low- and middle-income countries. Our primary goal was therefore to develop an alternative method to OCT for DME diagnosis by introducing spectral information derived from spontaneous electroretinogram (ERG) signals as a single input or combined with fundus that is much more widespread. Baseline ERGs were recorded in 233 patients and transformed into scalograms and spectrograms via Wavelet and Fourier transforms, respectively. Using transfer learning, distinct Convolutional Neural Networks (CNN) were trained as classifiers for DME using OCT, scalogram, spectrogram, and eye fundus images. Input data were randomly split into training and test sets with a proportion of 80 %-20 %, respectively. The top performers for each input type were selected, OpticNet-71 for OCT, DenseNet-201 for eye fundus, and non-evoked ERG-derived scalograms, to generate a combined model by assigning different weights for each of the selected models. Model validation was performed using a dataset alien to the training phase of the models. None of the models powered by mock ERG-derived input performed well. In contrast, hybrid models showed better results, in particular, the model powered by eye fundus combined with mock ERG-derived information with a 91 % AUC and 86 % F1-score, and the model powered by OCT and mock ERG-derived scalogram images with a 93 % AUC and 89 % F1-score. These data show that the spontaneous ERG-derived input adds predictive value to the fundus- and OCT-based models to diagnose DME, except for the sensitivity of the OCT model which remains the same. The inclusion of mock ERG signals, which have recently been shown to take only 5 min to record in daylight conditions, therefore represents a potential improvement over existing OCT-based models, as well as a reliable and cost-effective alternative when combined with the fundus, especially in underserved areas, to predict DME.

Development of a generative deep learning model to improve epiretinal membrane detection in fundus photography

BACKGROUND

The epiretinal membrane (ERM) is a common retinal disorder characterized by abnormal fibrocellular tissue at the vitreomacular interface. Most patients with ERM are asymptomatic at early stages. Therefore, screening for ERM will become increasingly important. Despite the high prevalence of ERM, few deep learning studies have investigated ERM detection in the color fundus photography (CFP) domain. In this study, we built a generative model to enhance ERM detection performance in the CFP.

METHODS

This deep learning study retrospectively collected 302 ERM and 1,250 healthy CFP data points from a healthcare center. The generative model using StyleGAN2 was trained using single-center data. EfficientNetB0 with StyleGAN2-based augmentation was validated using independent internal single-center data and external datasets. We randomly assigned healthcare center data to the development (80%) and internal validation (20%) datasets. Data from two publicly accessible sources were used as external validation datasets.

RESULTS

StyleGAN2 facilitated realistic CFP synthesis with the characteristic cellophane reflex features of the ERM. The proposed method with StyleGAN2-based augmentation outperformed the typical transfer learning without a generative adversarial network. The proposed model achieved an area under the receiver operating characteristic (AUC) curve of 0.926 for internal validation. AUCs of 0.951 and 0.914 were obtained for the two external validation datasets. Compared with the deep learning model without augmentation, StyleGAN2-based augmentation improved the detection performance and contributed to the focus on the location of the ERM.

CONCLUSIONS

We proposed an ERM detection model by synthesizing realistic CFP images with the pathological features of ERM through generative deep learning. We believe that our deep learning framework will help achieve a more accurate detection of ERM in a limited data setting.

Automated optic disk segmentation for optic disk edema classification using factorized gradient vector flow

One significant ocular symptom of neuro-ophthalmic disorders of the optic disk (OD) is optic disk edema (ODE). The etiologies of ODE are broad, with various symptoms and effects. Early detection of ODE can prevent potential vision loss and fatal vision problems. The texture of edematous OD significantly differs from the non-edematous OD in retinal images. As a result, techniques that usually work for non-edematous cases may not work well for edematous cases. We propose a fully automatic OD classification of edematous and non-edematous OD on fundus image collections containing a mixture of edematous and non-edematous ODs. The proposed algorithm involved localization, segmentation, and classification of edematous and non-edematous OD. The factorized gradient vector flow (FGVF) was used to segment the ODs. The OD type was classified using a linear support vector machine (SVM) based on 27 features extracted from the vessels, GLCM, color, and intensity line profile. The proposed method was tested on 295 images with 146 edematous cases and 149 non-edematous cases from three datasets. The segmentation achieves an average precision of 88.41%, recall of 89.35%, and F1-Score of 86.53%. The average classification accuracy is 99.40% and outperforms the state-of-the-art method by 3.43%.

Advances in medical image analysis with vision Transformers: A comprehensive review

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.

Deep-Ocular: Improved Transfer Learning Architecture Using Self-Attention and Dense Layers for Recognition of Ocular Diseases

It is difficult for clinicians or less-experienced ophthalmologists to detect early eye-related diseases. By hand, eye disease diagnosis is labor-intensive, prone to mistakes, and challenging because of the variety of ocular diseases such as glaucoma (GA), diabetic retinopathy (DR), cataract (CT), and normal eye-related diseases (NL). An automated ocular disease detection system with computer-aided diagnosis (CAD) tools is required to recognize eye-related diseases. Nowadays, deep learning (DL) algorithms enhance the classification results of retinograph images. To address these issues, we developed an intelligent detection system based on retinal fundus images. To create this system, we used ODIR and RFMiD datasets, which included various retinographics of distinct classes of the fundus, using cutting-edge image classification algorithms like ensemble-based transfer learning. In this paper, we suggest a three-step hybrid ensemble model that combines a classifier, a feature extractor, and a feature selector. The original image features are first extracted using a pre-trained AlexNet model with an enhanced structure. The improved AlexNet (iAlexNet) architecture with attention and dense layers offers enhanced feature extraction, task adaptability, interpretability, and potential accuracy benefits compared to other transfer learning architectures, making it particularly suited for tasks like retinograph classification. The extracted features are then selected using the ReliefF method, and then the most crucial elements are chosen to minimize the feature dimension. Finally, an XgBoost classifier offers classification outcomes based on the desired features. These classifications represent different ocular illnesses. We utilized data augmentation techniques to control class imbalance issues. The deep-ocular model, based mainly on the AlexNet-ReliefF-XgBoost model, achieves an accuracy of 95.13%. The results indicate the proposed ensemble model can assist dermatologists in making early decisions for the diagnosing and screening of eye-related diseases.

Discriminative ensemble meta-learning with co-regularization for rare fundus diseases diagnosis

Deep neural networks (DNNs) have been widely applied in the medical image community, contributing to automatic ophthalmic screening systems for some common diseases. However, the incidence of fundus diseases patterns exhibits a typical long-tailed distribution. In clinic, a small number of common fundus diseases have sufficient observed cases for large-scale analysis while most of the fundus diseases are infrequent. For these rare diseases with extremely low-data regimes, it is challenging to train DNNs to realize automatic diagnosis. In this work, we develop an automatic diagnosis system for rare fundus diseases, based on the meta-learning framework. The system incorporates a co-regularization loss and the ensemble-learning strategy into the meta-learning framework, fully leveraging the advantage of multi-scale hierarchical feature embedding. We initially conduct comparative experiments on our newly-constructed lightweight multi-disease fundus images dataset for the few-shot recognition task (namely, FundusData-FS). Moreover, we verify the cross-domain transferability from miniImageNet to FundusData-FS, and further confirm our method's good repeatability. Rigorous experiments demonstrate that our method can detect rare fundus diseases, and is superior to the state-of-the-art methods. These investigations demonstrate that the potential of our method for the real clinical practice is promising.

Generative Adversarial Networks in Medicine: Important Considerations for this Emerging Innovation in Artificial Intelligence

The advent of artificial intelligence (AI) and machine learning (ML) has revolutionized the field of medicine. Although highly effective, the rapid expansion of this technology has created some anticipated and unanticipated bioethical considerations. With these powerful applications, there is a necessity for framework regulations to ensure equitable and safe deployment of technology. Generative Adversarial Networks (GANs) are emerging ML techniques that have immense applications in medical imaging due to their ability to produce synthetic medical images and aid in medical AI training. Producing accurate synthetic images with GANs can address current limitations in AI development for medical imaging and overcome current dataset type and size constraints. Offsetting these constraints can dramatically improve the development and implementation of AI medical imaging and restructure the practice of medicine. As observed with its other AI predecessors, considerations must be taken into place to help regulate its development for clinical use. In this paper, we discuss the legal, ethical, and technical challenges for future safe integration of this technology in the healthcare sector.

Towards population-independent, multi-disease detection in fundus photographs

Independent validation studies of automatic diabetic retinopathy screening systems have recently shown a drop of screening performance on external data. Beyond diabetic retinopathy, this study investigates the generalizability of deep learning (DL) algorithms for screening various ocular anomalies in fundus photographs, across heterogeneous populations and imaging protocols. The following datasets are considered: OPHDIAT (France, diabetic population), OphtaMaine (France, general population), RIADD (India, general population) and ODIR (China, general population). Two multi-disease DL algorithms were developed: a Single-Dataset (SD) network, trained on the largest dataset (OPHDIAT), and a Multiple-Dataset (MD) network, trained on multiple datasets simultaneously. To assess their generalizability, both algorithms were evaluated whenever training and test data originate from overlapping datasets or from disjoint datasets. The SD network achieved a mean per-disease area under the receiver operating characteristic curve (mAUC) of 0.9571 on OPHDIAT. However, it generalized poorly to the other three datasets (mAUC < 0.9). When all four datasets were involved in training, the MD network significantly outperformed the SD network (p = 0.0058), indicating improved generality. However, in leave-one-dataset-out experiments, performance of the MD network was significantly lower on populations unseen during training than on populations involved in training (p < 0.0001), indicating imperfect generalizability.

Counteracting Data Bias and Class Imbalance-Towards a Useful and Reliable Retinal Disease Recognition System

Multiple studies presented satisfactory performances for the treatment of various ocular diseases. To date, there has been no study that describes a multiclass model, medically accurate, and trained on large diverse dataset. No study has addressed a class imbalance problem in one giant dataset originating from multiple large diverse eye fundus image collections. To ensure a real-life clinical environment and mitigate the problem of biased medical image data, 22 publicly available datasets were merged. To secure medical validity only Diabetic Retinopathy (DR), Age-Related Macular Degeneration (AMD) and Glaucoma (GL) were included. The state-of-the-art models ConvNext, RegNet and ResNet were utilized. In the resulting dataset, there were 86,415 normal, 3787 GL, 632 AMD and 34,379 DR fundus images. ConvNextTiny achieved the best results in terms of recognizing most of the examined eye diseases with the most metrics. The overall accuracy was 80.46 ± 1.48. Specific accuracy values were: 80.01 ± 1.10 for normal eye fundus, 97.20 ± 0.66 for GL, 98.14 ± 0.31 for AMD, 80.66 ± 1.27 for DR. A suitable screening model for the most prevalent retinal diseases in ageing societies was designed. The model was developed on a diverse, combined large dataset which made the obtained results less biased and more generalizable.

A Global Review of Publicly Available Datasets Containing Fundus Images: Characteristics, Barriers to Access, Usability, and Generalizability

This article provides a comprehensive and up-to-date overview of the repositories that contain color fundus images. We analyzed them regarding availability and legality, presented the datasets' characteristics, and identified labeled and unlabeled image sets. This study aimed to complete all publicly available color fundus image datasets to create a central catalog of available color fundus image datasets.

Retinal image enhancement based on color dominance of image

Real-time fundus images captured to detect multiple diseases are prone to different quality issues like illumination, noise, etc., resulting in less visibility of anomalies. So, enhancing the retinal fundus images is essential for a better prediction rate of eye diseases. In this paper, we propose Lab color space-based enhancement techniques for retinal image enhancement. Existing research works does not consider the relation between color spaces of the fundus image in selecting a specific channel to perform retinal image enhancement. Our unique contribution to this research work is utilizing the color dominance of an image in quantifying the distribution of information in the blue channel and performing enhancement in Lab space followed by a series of steps to optimize overall brightness and contrast. The test set of the Retinal Fundus Multi-disease Image Dataset is used to evaluate the performance of the proposed enhancement technique in identifying the presence or absence of retinal abnormality. The proposed technique achieved an accuracy of 89.53 percent.

TMM-Nets: Transferred Multi- to Mono-Modal Generation for Lupus Retinopathy Diagnosis

Rare diseases, which are severely underrepresented in basic and clinical research, can particularly benefit from machine learning techniques. However, current learning-based approaches usually focus on either mono-modal image data or matched multi-modal data, whereas the diagnosis of rare diseases necessitates the aggregation of unstructured and unmatched multi-modal image data due to their rare and diverse nature. In this study, we therefore propose diagnosis-guided multi-to-mono modal generation networks (TMM-Nets) along with training and testing procedures. TMM-Nets can transfer data from multiple sources to a single modality for diagnostic data structurization. To demonstrate their potential in the context of rare diseases, TMM-Nets were deployed to diagnose the lupus retinopathy (LR-SLE), leveraging unmatched regular and ultra-wide-field fundus images for transfer learning. The TMM-Nets encoded the transfer learning from diabetic retinopathy to LR-SLE based on the similarity of the fundus lesions. In addition, a lesion-aware multi-scale attention mechanism was developed for clinical alerts, enabling TMM-Nets not only to inform patient care, but also to provide insights consistent with those of clinicians. An adversarial strategy was also developed to refine multi- to mono-modal image generation based on diagnostic results and the data distribution to enhance the data augmentation performance. Compared to the baseline model, the TMM-Nets showed 35.19% and 33.56% F1 score improvements on the test and external validation sets, respectively. In addition, the TMM-Nets can be used to develop diagnostic models for other rare diseases.

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.

Detection of Glaucoma on Fundus Images Using Deep Learning on a New Image Set Obtained with a Smartphone and Handheld Ophthalmoscope

Statistics show that an estimated 64 million people worldwide suffer from glaucoma. To aid in the detection of this disease, this paper presents a new public dataset containing eye fundus images that was developed for glaucoma pattern-recognition studies using deep learning (DL). The dataset, denoted Brazil Glaucoma, comprises 2000 images obtained from 1000 volunteers categorized into two groups: those with glaucoma (50%) and those without glaucoma (50%). All images were captured with a smartphone attached to a Welch Allyn panoptic direct ophthalmoscope. Further, a DL approach for the automatic detection of glaucoma was developed using the new dataset as input to a convolutional neural network ensemble model. The accuracy between positive and negative glaucoma detection, sensitivity, and specificity were calculated using five-fold cross-validation to train and refine the classification model. The results showed that the proposed method can identify glaucoma from eye fundus images with an accuracy of 90.0%. Thus, the combination of fundus images obtained using a smartphone attached to a portable panoptic ophthalmoscope and artificial intelligence algorithms yielded satisfactory results in the overall accuracy of glaucoma detection tests. Consequently, the proposed approach can contribute to the development of technologies aimed at massive population screening of the disease.

DeepPDT-Net: predicting the outcome of photodynamic therapy for chronic central serous chorioretinopathy using two-stage multimodal transfer learning

Central serous chorioretinopathy (CSC), characterized by serous detachment of the macular retina, can cause permanent vision loss in the chronic course. Chronic CSC is generally treated with photodynamic therapy (PDT), which is costly and quite invasive, and the results are unpredictable. In a retrospective case-control study design, we developed a two-stage deep learning model to predict 1-year outcome of PDT using initial multimodal clinical data. The training dataset included 166 eyes with chronic CSC and an additional learning dataset containing 745 healthy control eyes. A pre-trained ResNet50-based convolutional neural network was first trained with normal fundus photographs (FPs) to detect CSC and then adapted to predict CSC treatability through transfer learning. The domain-specific ResNet50 successfully predicted treatable and refractory CSC (accuracy, 83.9%). Then other multimodal clinical data were integrated with the FP deep features using XGBoost.The final combined model (DeepPDT-Net) outperformed the domain-specific ResNet50 (accuracy, 88.0%). The FP deep features had the greatest impact on DeepPDT-Net performance, followed by central foveal thickness and age. In conclusion, DeepPDT-Net could solve the PDT outcome prediction task challenging even to retinal specialists. This two-stage strategy, adopting transfer learning and concatenating multimodal data, can overcome the clinical prediction obstacles arising from insufficient datasets.

Deep Ensemble Learning for Retinal Image Classification

Purpose

Vision impairment affects 2.2 billion people worldwide, half of which is preventable with early detection and treatment. Currently, automatic screening of ocular pathologies using convolutional neural networks (CNNs) on retinal fundus photographs is limited to a few pathologies. Simultaneous detection of multiple ophthalmic pathologies would increase clinical usability and uptake.

Methods

Two thousand five hundred sixty images were used from the Retinal Fundus Multi-Disease Image Dataset (RFMiD). Models were trained (n = 1920) and validated (n = 640). Five selected CNN architectures were trained to predict the presence of any pathology and categorize the 28 pathologies. All models were trained to minimize asymmetric loss, a modified form of binary cross-entropy. Individual model predictions were averaged to obtain a final ensembled model and assessed for mean area under the receiver-operator characteristic curve (AUROC) for disease screening (healthy versus pathologic image) and classification (AUROC for each class).

Results

The ensemble network achieved a disease screening (healthy versus pathologic) AUROC score of 0.9613. The highest single network score was 0.9586 using the SE-ResNeXt architecture. For individual disease classification, the average AUROC score for each class was 0.9295.

Conclusions

Retinal fundus images analyzed by an ensemble of CNNs trained to minimize asymmetric loss were effective in detection and classification of ocular pathologies than individual models. External validation is needed to translate machine learning models to diverse clinical contexts.

Translational Relevance

This study demonstrates the potential benefit of ensemble-based deep learning methods on improving automatic screening and diagnosis of multiple ocular pathologies from fundoscopy imaging.

Deep learning methods hold promise for light fluence compensation in three-dimensional optoacoustic imaging

SIGNIFICANCE

Quantitative optoacoustic imaging (QOAI) continues to be a challenge due to the influence of nonlinear optical fluence distribution, which distorts the optoacoustic image representation. Nonlinear optical fluence correction in OA imaging is highly ill-posed, leading to the inaccurate recovery of optical absorption maps. This work aims to recover the optical absorption maps using deep learning (DL) approach by correcting for the fluence effect.

AIM

Different DL models were compared and investigated to enable optical absorption coefficient recovery at a particular wavelength in a nonhomogeneous foreground and background medium.

APPROACH

Data-driven models were trained with two-dimensional (2D) Blood vessel and three-dimensional (3D) numerical breast phantom with highly heterogeneous/realistic structures to correct for the nonlinear optical fluence distribution. The trained DL models such as U-Net, Fully Dense (FD) U-Net, Y-Net, FD Y-Net, Deep residual U-Net (Deep ResU-Net), and generative adversarial network (GAN) were tested to evaluate the performance of optical absorption coefficient recovery (or fluence compensation) with in-silico and in-vivo datasets.

RESULTS

The results indicated that FD U-Net-based deconvolution improves by about 10% over reconstructed optoacoustic images in terms of peak-signal-to-noise ratio. Further, it was observed that DL models can indeed highlight deep-seated structures with higher contrast due to fluence compensation. Importantly, the DL models were found to be about 17 times faster than solving diffusion equation for fluence correction.

CONCLUSIONS

The DL methods were able to compensate for nonlinear optical fluence distribution more effectively and improve the optoacoustic image quality.

Retinal Vessel Segmentation With Skeletal Prior and Contrastive Loss

The morphology of retinal vessels is closely associated with many kinds of ophthalmic diseases. Although huge progress in retinal vessel segmentation has been achieved with the advancement of deep learning, some challenging issues remain. For example, vessels can be disturbed or covered by other components presented in the retina (such as optic disc or lesions). Moreover, some thin vessels are also easily missed by current methods. In addition, existing fundus image datasets are generally tiny, due to the difficulty of vessel labeling. In this work, a new network called SkelCon is proposed to deal with these problems by introducing skeletal prior and contrastive loss. A skeleton fitting module is developed to preserve the morphology of the vessels and improve the completeness and continuity of thin vessels. A contrastive loss is employed to enhance the discrimination between vessels and background. In addition, a new data augmentation method is proposed to enrich the training samples and improve the robustness of the proposed model. Extensive validations were performed on several popular datasets (DRIVE, STARE, CHASE, and HRF), recently developed datasets (UoA-DR, IOSTAR, and RC-SLO), and some challenging clinical images (from RFMiD and JSIEC39 datasets). In addition, some specially designed metrics for vessel segmentation, including connectivity, overlapping area, consistency of vessel length, revised sensitivity, specificity, and accuracy were used for quantitative evaluation. The experimental results show that, the proposed model achieves state-of-the-art performance and significantly outperforms compared methods when extracting thin vessels in the regions of lesions or optic disc. Source code is available at https://www.github.com/tyb311/SkelCon.

Multi-Expert Deep Networks for Multi-Disease Detection in Retinal Fundus Images

Automatic diagnosis of eye diseases from retinal fundus images is quite challenging. Common public datasets include images of subjects with multiple diseases with uneven distribution of labels. Rare diseases are especially challenging due to their under-representation in such datasets. In this paper, we propose a training pipeline for the multi-labeled classification with uneven distribution of the sample size and sample difficulty. First, we guide the training of the initial model by weighing the training loss using an inverse-frequency for each class. This will balance the training on over-represented and under-represented samples. We then adjust the class weights using the aggregated loss for each class, and train for more iterations. In this way, the model at each iteration will focus more on difficult samples and cover the shortcomings of the previous model. Finally, we ensemble together all the models using out proposed Heuristic Stacking algorithm for improving multi-label predictions beyond simple averaging. Our experimental results on the Retinal Image Analysis for Multi-Disease Detection(RIADD)-2021 challenge dataset show that the proposed approach achieves a 88.24 % accuracy score, which is competitive with the top three ranked methods of the competition. Furthermore, we perform ablation study to stress-test our Heuristic Stacking ensemble methods versus classical methods such as bagging n multi-label classification problems.

Machine learning predicting myopic regression after corneal refractive surgery using preoperative data and fundus photography

PURPOSE

Myopic regression after surgery is the most common long-term complication of refractive surgery, but it is difficult to identify myopic regression without long-term observation. This study aimed to develop machine learning models to identify high-risk patients for refractive regression based on preoperative data and fundus photography.

METHODS

This retrospective study assigned subjects to the training (n = 1606 eyes) and validation (n = 403 eyes) datasets with chronological data splitting. Machine learning models with ResNet50 (for image analysis) and XGBoost (for integration of all variables and fundus photography) were developed based on subjects who underwent corneal refractive surgery. The primary outcome was the predictive performance for the presence of myopic regression at 4 years of follow-up examination postoperatively.

RESULTS

By integrating all factors and fundus photography, the final combined machine learning model showed good performance to predict myopic regression of more than 0.5 D (area under the receiver operating characteristic curve [ROC-AUC], 0.753; 95% confidence interval [CI], 0.710-0.793). The performance of the final model was better than the single ResNet50 model only using fundus photography (ROC-AUC, 0.673; 95% CI, 0.627-0.716). The top-five most important input features were fundus photography, preoperative anterior chamber depth, planned ablation thickness, age, and preoperative central corneal thickness.

CONCLUSION

Our machine learning algorithm provides an efficient strategy to identify high-risk patients with myopic regression without additional labor, cost, and time. Surgeons might benefit from preoperative risk assessment of myopic regression, patient counseling before surgery, and surgical option decisions.

Multiple Ocular Disease Diagnosis Using Fundus Images Based on Multi-Label Deep Learning Classification

Designing computer-aided diagnosis (CAD) systems that can automatically detect ocular diseases (ODs) has become an active research field in the health domain. Although the human eye might have more than one OD simultaneously, most existing systems are designed to detect specific eye diseases. Therefore, it is crucial to develop new CAD systems that can detect multiple ODs simultaneously. This paper presents a novel multi-label convolutional neural network (ML-CNN) system based on ML classification (MLC) to diagnose various ODs from color fundus images. The proposed ML-CNN-based system consists of three main phases: the preprocessing phase, which includes normalization and augmentation using several transformation processes, the modeling phase, and the prediction phase. The proposed ML-CNN consists of three convolution (CONV) layers and one max pooling (MP) layer. Then, two CONV layers are performed, followed by one MP and dropout (DO). After that, one flatten layer is performed, followed by one fully connected (FC) layer. We added another DO once again, and finally, one FC layer with 45 nodes is performed. The system outputs the probabilities of all 45 diseases in each image. We validated the model by using cross-validation (CV) and measured the performance by five different metrics: accuracy (ACC), recall, precision, Dice similarity coefficient (DSC), and area under the curve (AUC). The results are 94.3%, 80%, 91.5%, 99%, and 96.7%, respectively. The comparisons with the existing built-in models, such as MobileNetV2, DenseNet201, SeResNext50, InceptionV3, and InceptionresNetv2, demonstrate the superiority of the proposed ML-CNN model.

A deep learning approach for detection of shallow anterior chamber depth based on the hidden features of fundus photographs

BACKGROUND AND OBJECTIVES

Patients with angle-closure glaucoma (ACG) are asymptomatic until they experience a painful attack. Shallow anterior chamber depth (ACD) is considered a significant risk factor for ACG. We propose a deep learning approach to detect shallow ACD using fundus photographs and to identify the hidden features of shallow ACD.

METHODS

This retrospective study assigned healthy subjects to the training (n = 1188 eyes) and test (n = 594) datasets (prospective validation design). We used a deep learning approach to estimate ACD and build a classification model to identify eyes with a shallow ACD. The proposed method, including subtraction of the input and output images of CycleGAN and a thresholding algorithm, was adopted to visualize the characteristic features of fundus photographs with a shallow ACD.

RESULTS

The deep learning model integrating fundus photographs and clinical variables achieved areas under the receiver operating characteristic curve of 0.978 (95% confidence interval [CI], 0.963-0.988) for an ACD ≤ 2.60 mm and 0.895 (95% CI, 0.868-0.919) for an ACD ≤ 2.80 mm, and outperformed the regression model using only clinical variables. However, the difference between shallow and deep ACD classes on fundus photographs was difficult to be detected with the naked eye. We were unable to identify the features of shallow ACD using the Grad-CAM. The CycleGAN-based feature images showed that area around the macula and optic disk significantly contributed to the classification of fundus photographs with a shallow ACD.

CONCLUSIONS

We demonstrated the feasibility of a novel deep learning model to detect a shallow ACD as a screening tool for ACG using fundus photographs. The CycleGAN-based feature map showed the hidden characteristic features of shallow ACD that were previously undetectable by conventional techniques and ophthalmologists. This framework will facilitate the early detection of shallow ACD to prevent overlooking the risks associated with ACG.

Peripapillary Atrophy Segmentation and Classification Methodologies for Glaucoma Image Detection: A Review

Information-based image processing and computer vision methods are utilized in several healthcare organizations to diagnose diseases. The irregularities in the visual system are identified over fundus images shaped over a fundus camera. Among ophthalmology diseases, glaucoma is measured as the most common case that can lead to neurodegenerative illness. The unsuitable fluid pressure inside the eye within the visual system is described as the major cause of those diseases. Glaucoma has no symptoms in the early stages, and if it is not treated, it may result in total blindness. Diagnosing glaucoma at an early stage may prevent permanent blindness. Manual inspection of the human eye may be a solution, but it depends on the skills of the individuals involved. The auto diagnosis of glaucoma by applying a consolidation of computer vision, artificial intelligence, and image processing can aid in the ban and detection of those diseases. In this review article, we aim to introduce a review of the numerous approaches based on peripapillary atrophy segmentation and classification that can detect these diseases, as well as details about the publicly available image benchmarks, datasets, and measurement of performance. The review article introduces the demonstrated research of numerous available study models that objectively diagnose glaucoma via peripapillary atrophy from the lowest level of feature extraction to the current direction based on deep learning. The advantages and disadvantages of each method are addressed in detail, and tabular descriptions are included to highlight the results of each category. Moreover, the frameworks of each approach and fundus image datasets are provided. The improved reporting of our study would help in providing possible future work directions to diagnose glaucoma in conclusion.

Simple Code Implementation for Deep Learning-Based Segmentation to Evaluate Central Serous Chorioretinopathy in Fundus Photography

Purpose

Central serous chorioretinopathy (CSC) is a retinal disease that frequently shows resolution and recurrence with serous detachment of the neurosensory retina. Here, we present a deep learning analysis of subretinal fluid (SRF) lesion segmentation in fundus photographs to evaluate CSC.

Methods

We collected 194 fundus photographs of SRF lesions from the patients with CSC. Three graders manually annotated of the entire SRF area in the retinal images. The dataset was randomly separated into training (90%) and validation (10%) datasets. We used the U-Net segmentation model based on conditional generative adversarial networks (pix2pix) to detect the SRF lesions. The algorithms were trained and validated using Google Colaboratory. Researchers did not need prior knowledge of coding skills or computing resources to implement this code.

Results

The validation results showed that the Jaccard index and Dice coefficient scores were 0.619 and 0.763, respectively. In most cases, the segmentation results overlapped with most of the reference areas in the annotated images. However, cases with exceptional SRFs were not accurate in terms of prediction. Using Colaboratory, the proposed segmentation task ran easily in a web-based environment without setup or personal computing resources.

Conclusions

The results suggest that the deep learning model based on U-Net from the pix2pix algorithm is suitable for the automatic segmentation of SRF lesions to evaluate CSC.

Translational Relevance

Our code implementation has the potential to facilitate ophthalmology research; in particular, deep learning–based segmentation can assist in the development of pathological lesion detection solutions.