Assessment of Deep Generative Models for High-Resolution Synthetic Retinal Image Generation of Age-Related Macular Degeneration

Improving Artificial Intelligence-based Microbial Keratitis Screening Tools Constrained by Limited Data Using Synthetic Generation of Slit-Lamp Photos

Objective

We developed a novel slit-lamp photography (SLP) generative adversarial network (GAN) model using limited data to supplement and improve the performance of an artificial intelligence (AI)-based microbial keratitis (MK) screening model.

Design

Cross-sectional study.

Subjects

Slit-lamp photographs of 67 healthy and 36 MK eyes were prospectively and retrospectively collected at a tertiary care ophthalmology clinic at a large academic institution.

Methods

We trained the GAN model StyleGAN2-ADA on healthy and MK SLPs to generate synthetic images. To assess synthetic image quality, we performed a visual Turing test. Three cornea fellows tested their ability to identify 20 images each of (1) real healthy, (2) real diseased, (3) synthetic healthy, and (4) synthetic diseased. We also used Kernel Inception Distance (KID) to quantitatively measure realism and variation of synthetic images. Using the same dataset used to train the GAN model, we trained 2 DenseNet121 AI models to grade SLP images as healthy or MK with (1) only real images and (2) real supplemented with GAN-generated images.

Main Outcome Measures

Classification performance of MK screening models trained with only real images compared to a model trained with both limited real and supplemented synthetic GAN images.

Results

For the visual Turing test, the fellows on average rated synthetic images as good quality (83.3% ± 12.0% of images), and synthetic and real images were found to depict pertinent anatomy and pathology for accurate classification (96.3% ± 2.19% of images). These experts could distinguish between real and synthetic images (accuracy: 92.5% ± 9.01%). Analysis of KID score for synthetic images indicated realism and variation. The MK screening model trained on both limited real and supplemented synthetic data (area under the receiver-operator characteristic curve: 0.93, bootstrapping 95% CI: 0.77-1.0) outperformed the model trained with only real data (area under the receiver-operator characteristic curve: 0.76, 95% CI: 0.50-1.0), with an improvement of 0.17 (95% CI: 0-0.4; 2-tailed t test P = 0.076).

Conclusions

Artificial intelligence-based MK classification may be improved by supplementation of limited real training data with synthetic data generated by GANs.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

A data-efficient strategy for building high-performing medical foundation models

Foundation models are pretrained on massive datasets. However, collecting medical datasets is expensive and time-consuming, and raises privacy concerns. Here we show that synthetic data generated via conditioning with disease labels can be leveraged for building high-performing medical foundation models. We pretrained a retinal foundation model, first with approximately one million synthetic retinal images with physiological structures and feature distribution consistent with real counterparts, and then with only 16.7% of the 904,170 real-world colour fundus photography images required in a recently reported retinal foundation model (RETFound). The data-efficient model performed as well or better than RETFound across nine public datasets and four diagnostic tasks; and for diabetic-retinopathy grading, it used only 40% of the expert-annotated training data used by RETFound. We also support the generalizability of the data-efficient strategy by building a classifier for the detection of tuberculosis on chest X-ray images. The text-conditioned generation of synthetic data may enhance the performance and generalization of medical foundation models.

AI image generation technology in ophthalmology: Use, misuse and future applications

BACKGROUND

AI-powered image generation technology holds the potential to reshape medical practice, yet it remains an unfamiliar technology for both medical researchers and clinicians alike. Given the adoption of this technology relies on clinician understanding and acceptance, we sought to demystify its use in ophthalmology. To this end, we present a literature review on image generation technology in ophthalmology, examining both its theoretical applications and future role in clinical practice.

METHODS

First, we consider the key model designs used for image synthesis, including generative adversarial networks, autoencoders, and diffusion models. We then perform a survey of the literature for image generation technology in ophthalmology prior to September 2024, presenting both the type of model used and its clinical application. Finally, we discuss the limitations of this technology, the risks of its misuse and the future directions of research in this field.

RESULTS

Applications of this technology include improving AI diagnostic models, inter-modality image transformation, more accurate treatment and disease prognostication, image denoising, and individualised education. Key barriers to its adoption include bias in generative models, risks to patient data security, computational and logistical barriers to development, challenges with model explainability, inconsistent use of validation metrics between studies and misuse of synthetic images. Looking forward, researchers are placing a further emphasis on clinically grounded metrics, the development of image generation foundation models and the implementation of methods to ensure data provenance.

CONCLUSION

Compared to other medical applications of AI, image generation is still in its infancy. Yet, it holds the potential to revolutionise ophthalmology across research, education and clinical practice. This review aims to guide ophthalmic researchers wanting to leverage this technology, while also providing an insight for clinicians on how it may change ophthalmic practice in the future.

Realistic fundus photograph generation for improving automated disease classification

AIMS

This study aims to investigate whether denoising diffusion probabilistic models (DDPMs) could generate realistic retinal images, and if they could be used to improve the performance of a deep convolutional neural network (CNN) ensemble for multiple retinal disease classification, which was previously shown to outperform human experts.

METHODS

We trained DDPMs to generate retinal fundus images representing diabetic retinopathy, age-related macular degeneration, glaucoma or normal eyes. Eight board-certified ophthalmologists evaluated 96 test images to assess the realism of generated images and classified them based on disease labels. Subsequently, between 100 and 1000 generated images were employed to augment training of deep convolutional ensembles for classifying retinal disease. We measured the accuracy of ophthalmologists in correctly identifying real and generated images. We also measured the classification accuracy, F-score and area under the receiver operating curve of a trained CNN in classifying retinal diseases from a test set of 100 fundus images.

RESULTS

Ophthalmologists exhibited a mean accuracy of 61.1% (range: 51.0%-68.8%) in differentiating real and generated images. Augmenting the training set with 238 generated images in the smallest class statistically significantly improved the F-score and accuracy by 5.3% and 5.8%, respectively (p<0.01) in a retinal disease classification task, compared with a baseline model trained only with real images.

CONCLUSIONS

Latent diffusion models generated highly realistic retinal images, as validated by human experts. Adding generated images to the training set improved performance of a CNN ensemble without requiring additional real patient data.

Generative artificial intelligence in ophthalmology

Generative artificial intelligence (AI) has revolutionized medicine over the past several years. A generative adversarial network (GAN) is a deep learning framework that has become a powerful technique in medicine, particularly in ophthalmology for image analysis. In this paper we review the current ophthalmic literature involving GANs, and highlight key contributions in the field. We briefly touch on ChatGPT, another application of generative AI, and its potential in ophthalmology. We also explore the potential uses for GANs in ocular imaging, with a specific emphasis on 3 primary domains: image enhancement, disease identification, and generating of synthetic data. PubMed, Ovid MEDLINE, Google Scholar were searched from inception to October 30, 2022, to identify applications of GAN in ophthalmology. A total of 40 papers were included in this review. We cover various applications of GANs in ophthalmic-related imaging including optical coherence tomography, orbital magnetic resonance imaging, fundus photography, and ultrasound; however, we also highlight several challenges that resulted in the generation of inaccurate and atypical results during certain iterations. Finally, we examine future directions and considerations for generative AI in ophthalmology.

Three-Dimensional Bone-Image Synthesis with Generative Adversarial Networks

Medical image processing has been highlighted as an area where deep-learning-based models have the greatest potential. However, in the medical field, in particular, problems of data availability and privacy are hampering research progress and, thus, rapid implementation in clinical routine. The generation of synthetic data not only ensures privacy but also allows the drawing of new patients with specific characteristics, enabling the development of data-driven models on a much larger scale. This work demonstrates that three-dimensional generative adversarial networks (GANs) can be efficiently trained to generate high-resolution medical volumes with finely detailed voxel-based architectures. In addition, GAN inversion is successfully implemented for the three-dimensional setting and used for extensive research on model interpretability and applications such as image morphing, attribute editing, and style mixing. The results are comprehensively validated on a database of three-dimensional HR-pQCT instances representing the bone micro-architecture of the distal radius.

Synthetic Data Generation via Generative Adversarial Networks in Healthcare: A Systematic Review of Image- and Signal-Based Studies

Generative Adversarial Networks (GANs) have emerged as a powerful tool in artificial intelligence, particularly for unsupervised learning. This systematic review analyzes GAN applications in healthcare, focusing on image and signal-based studies across various clinical domains. Following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines, we reviewed 72 relevant journal articles. Our findings reveal that magnetic resonance imaging (MRI) and electrocardiogram (ECG) signal acquisition techniques were most utilized, with brain studies (22%), cardiology (18%), cancer (15%), ophthalmology (12%), and lung studies (10%) being the most researched areas. We discuss key GAN architectures, including cGAN (31%) and CycleGAN (18%), along with datasets, evaluation metrics, and performance outcomes. The review highlights promising data augmentation, anonymization, and multi-task learning results. We identify current limitations, such as the lack of standardized metrics and direct comparisons, and propose future directions, including the development of no-reference metrics, immersive simulation scenarios, and enhanced interpretability.

Vision language models in ophthalmology

PURPOSE OF REVIEW

Vision Language Models are an emerging paradigm in artificial intelligence that offers the potential to natively analyze both image and textual data simultaneously, within a single model. The fusion of these two modalities is of particular relevance to ophthalmology, which has historically involved specialized imaging techniques such as angiography, optical coherence tomography, and fundus photography, while also interfacing with electronic health records that include free text descriptions. This review then surveys the fast-evolving field of Vision Language Models as they apply to current ophthalmologic research and practice.

RECENT FINDINGS

Although models incorporating both image and text data have a long provenance in ophthalmology, effective multimodal Vision Language Models are a recent development exploiting advances in technologies such as transformer and autoencoder models.

SUMMARY

Vision Language Models offer the potential to assist and streamline the existing clinical workflow in ophthalmology, whether previsit, during, or post-visit. There are, however, also important challenges to be overcome, particularly regarding patient privacy and explainability of model recommendations.

Artificial intelligence for diagnosing exudative age-related macular degeneration

BACKGROUND

Age-related macular degeneration (AMD) is a retinal disorder characterized by central retinal (macular) damage. Approximately 10% to 20% of non-exudative AMD cases progress to the exudative form, which may result in rapid deterioration of central vision. Individuals with exudative AMD (eAMD) need prompt consultation with retinal specialists to minimize the risk and extent of vision loss. Traditional methods of diagnosing ophthalmic disease rely on clinical evaluation and multiple imaging techniques, which can be resource-consuming. Tests leveraging artificial intelligence (AI) hold the promise of automatically identifying and categorizing pathological features, enabling the timely diagnosis and treatment of eAMD.

OBJECTIVES

To determine the diagnostic accuracy of artificial intelligence (AI) as a triaging tool for exudative age-related macular degeneration (eAMD).

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, three clinical trials registries, and Data Archiving and Networked Services (DANS) for gray literature. We did not restrict searches by language or publication date. The date of the last search was April 2024.

SELECTION CRITERIA

Included studies compared the test performance of algorithms with that of human readers to detect eAMD on retinal images collected from people with AMD who were evaluated at eye clinics in community or academic medical centers, and who were not receiving treatment for eAMD when the images were taken. We included algorithms that were either internally or externally validated or both.

DATA COLLECTION AND ANALYSIS

Pairs of review authors independently extracted data and assessed study quality using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool with revised signaling questions. For studies that reported more than one set of performance results, we extracted only one set of diagnostic accuracy data per study based on the last development stage or the optimal algorithm as indicated by the study authors. For two-class algorithms, we collected data from the 2x2 table whenever feasible. For multi-class algorithms, we first consolidated data from all classes other than eAMD before constructing the corresponding 2x2 tables. Assuming a common positivity threshold applied by the included studies, we chose random-effects, bivariate logistic models to estimate summary sensitivity and specificity as the primary performance metrics.

MAIN RESULTS

We identified 36 eligible studies that reported 40 sets of algorithm performance data, encompassing over 16,000 participants and 62,000 images. We included 28 studies (78%) that reported 31 algorithms with performance data in the meta-analysis. The remaining nine studies (25%) reported eight algorithms that lacked usable performance data; we reported them in the qualitative synthesis. Study characteristics and risk of bias Most studies were conducted in Asia, followed by Europe, the USA, and collaborative efforts spanning multiple countries. Most studies identified study participants from the hospital setting, while others used retinal images from public repositories; a few studies did not specify image sources. Based on four of the 36 studies reporting demographic information, the age of the study participants ranged from 62 to 82 years. The included algorithms used various retinal image types as model input, such as optical coherence tomography (OCT) images (N = 15), fundus images (N = 6), and multi-modal imaging (N = 7). The predominant core method used was deep neural networks. All studies that reported externally validated algorithms were at high risk of bias mainly due to potential selection bias from either a two-gate design or the inappropriate exclusion of potentially eligible retinal images (or participants). Findings Only three of the 40 included algorithms were externally validated (7.5%, 3/40). The summary sensitivity and specificity were 0.94 (95% confidence interval (CI) 0.90 to 0.97) and 0.99 (95% CI 0.76 to 1.00), respectively, when compared to human graders (3 studies; 27,872 images; low-certainty evidence). The prevalence of images with eAMD ranged from 0.3% to 49%. Twenty-eight algorithms were reportedly either internally validated (20%, 8/40) or tested on a development set (50%, 20/40); the pooled sensitivity and specificity were 0.93 (95% CI 0.89 to 0.96) and 0.96 (95% CI 0.94 to 0.98), respectively, when compared to human graders (28 studies; 33,409 images; low-certainty evidence). We did not identify significant sources of heterogeneity among these 28 algorithms. Although algorithms using OCT images appeared more homogeneous and had the highest summary specificity (0.97, 95% CI 0.93 to 0.98), they were not superior to algorithms using fundus images alone (0.94, 95% CI 0.89 to 0.97) or multimodal imaging (0.96, 95% CI 0.88 to 0.99; P for meta-regression = 0.239). The median prevalence of images with eAMD was 30% (interquartile range [IQR] 22% to 39%). We did not include eight studies that described nine algorithms (one study reported two sets of algorithm results) to distinguish eAMD from normal images, images of other AMD, or other non-AMD retinal lesions in the meta-analysis. Five of these algorithms were generally based on smaller datasets (range 21 to 218 participants per study) yet with a higher prevalence of eAMD images (range 33% to 66%). Relative to human graders, the reported sensitivity in these studies ranged from 0.95 and 0.97, while the specificity ranged from 0.94 to 0.99. Similarly, using small datasets (range 46 to 106), an additional four algorithms for detecting eAMD from other retinal lesions showed high sensitivity (range 0.96 to 1.00) and specificity (range 0.77 to 1.00).

AUTHORS' CONCLUSIONS

Low- to very low-certainty evidence suggests that an algorithm-based test may correctly identify most individuals with eAMD without increasing unnecessary referrals (false positives) in either the primary or the specialty care settings. There were significant concerns for applying the review findings due to variations in the eAMD prevalence in the included studies. In addition, among the included algorithm-based tests, diagnostic accuracy estimates were at risk of bias due to study participants not reflecting real-world characteristics, inadequate model validation, and the likelihood of selective results reporting. Limited quality and quantity of externally validated algorithms highlighted the need for high-certainty evidence. This evidence will require a standardized definition for eAMD on different imaging modalities and external validation of the algorithm to assess generalizability.

VSG-GAN: A high-fidelity image synthesis method with semantic manipulation in retinal fundus image

In recent years, advancements in retinal image analysis, driven by machine learning and deep learning techniques, have enhanced disease detection and diagnosis through automated feature extraction. However, challenges persist, including limited data set diversity due to privacy concerns and imbalanced sample pairs, hindering effective model training. To address these issues, we introduce the vessel and style guided generative adversarial network (VSG-GAN), an innovative algorithm building upon the foundational concept of GAN. In VSG-GAN, a generator and discriminator engage in an adversarial process to produce realistic retinal images. Our approach decouples retinal image generation into distinct modules: the vascular skeleton and background style. Leveraging style transformation and GAN inversion, our proposed hierarchical variational autoencoder module generates retinal images with diverse morphological traits. In addition, the spatially adaptive denormalization module ensures consistency between input and generated images. We evaluate our model on MESSIDOR and RITE data sets using various metrics, including structural similarity index measure, inception score, Fréchet inception distance, and kernel inception distance. Our results demonstrate the superiority of VSG-GAN, outperforming existing methods across all evaluation assessments. This underscores its effectiveness in addressing data set limitations and imbalances. Our algorithm provides a novel solution to challenges in retinal image analysis by offering diverse and realistic retinal image generation. Implementing the VSG-GAN augmentation approach on downstream diabetic retinopathy classification tasks has shown enhanced disease diagnosis accuracy, further advancing the utility of machine learning in this domain.

Generative Artificial Intelligence Enhancements for Reducing Image-based Training Data Requirements

Objective

Training data fuel and shape the development of artificial intelligence (AI) models. Intensive data requirements are a major bottleneck limiting the success of AI tools in sectors with inherently scarce data. In health care, training data are difficult to curate, triggering growing concerns that the current lack of access to health care by under-privileged social groups will translate into future bias in health care AIs. In this report, we developed an autoencoder to grow and enhance inherently scarce datasets to alleviate our dependence on big data.

Design

Computational study with open-source data.

Subjects

The data were obtained from 6 open-source datasets comprising patients aged 40-80 years in Singapore, China, India, and Spain.

Methods

The reported framework generates synthetic images based on real-world patient imaging data. As a test case, we used autoencoder to expand publicly available training sets of optic disc photos, and evaluated the ability of the resultant datasets to train AI models in the detection of glaucomatous optic neuropathy.

Main Outcome Measures

Area under the receiver operating characteristic curve (AUC) were used to evaluate the performance of the glaucoma detector. A higher AUC indicates better detection performance.

Results

Results show that enhancing datasets with synthetic images generated by autoencoder led to superior training sets that improved the performance of AI models.

Conclusions

Our findings here help address the increasingly untenable data volume and quality requirements for AI model development and have implications beyond health care, toward empowering AI adoption for all similarly data-challenged fields.

Financial Disclosures

The authors have no proprietary or commercial interest in any materials discussed in this article.

Generative adversarial networks synthetic optical coherence tomography images as an education tool for image diagnosis of macular diseases: a randomized trial

Purpose

This study aimed to evaluate the effectiveness of generative adversarial networks (GANs) in creating synthetic OCT images as an educational tool for teaching image diagnosis of macular diseases to medical students and ophthalmic residents.

Methods

In this randomized trial, 20 fifth-year medical students and 20 ophthalmic residents were enrolled and randomly assigned (1:1 allocation) into Group real OCT and Group GANs OCT. All participants had a pretest to assess their educational background, followed by a 30-min smartphone-based education program using GANs or real OCT images for macular disease recognition training. Two additional tests were scheduled: one 5 min after the training to assess short-term performance, and another 1 week later to assess long-term performance. Scores and time consumption were recorded and compared. After all the tests, participants completed an anonymous subjective questionnaire.

Results

Group GANs OCT scores increased from 80.0 (46.0 to 85.5) to 92.0 (81.0 to 95.5) 5 min after training (p < 0.001) and 92.30 ± 5.36 1 week after training (p < 0.001). Similarly, Group real OCT scores increased from 66.00 ± 19.52 to 92.90 ± 5.71 (p < 0.001), respectively. When compared between two groups, no statistically significant difference was found in test scores, score improvements, or time consumption. After training, medical students had a significantly higher score improvement than residents (p < 0.001).

Conclusion

The education tool using synthetic OCT images had a similar educational ability compared to that using real OCT images, which improved the interpretation ability of ophthalmic residents and medical students in both short-term and long-term performances. The smartphone-based educational tool could be widely promoted for educational applications.Clinical trial registration: https://www.chictr.org.cn, Chinese Clinical Trial Registry [No. ChiCTR 2100053195].

Latest developments of generative artificial intelligence and applications in ophthalmology

The emergence of generative artificial intelligence (AI) has revolutionized various fields. In ophthalmology, generative AI has the potential to enhance efficiency, accuracy, personalization and innovation in clinical practice and medical research, through processing data, streamlining medical documentation, facilitating patient-doctor communication, aiding in clinical decision-making, and simulating clinical trials. This review focuses on the development and integration of generative AI models into clinical workflows and scientific research of ophthalmology. It outlines the need for development of a standard framework for comprehensive assessments, robust evidence, and exploration of the potential of multimodal capabilities and intelligent agents. Additionally, the review addresses the risks in AI model development and application in clinical service and research of ophthalmology, including data privacy, data bias, adaptation friction, over interdependence, and job replacement, based on which we summarized a risk management framework to mitigate these concerns. This review highlights the transformative potential of generative AI in enhancing patient care, improving operational efficiency in the clinical service and research in ophthalmology. It also advocates for a balanced approach to its adoption.

Retinal vascular morphological characteristics in diabetic retinopathy: an artificial intelligence study using a transfer learning system to analyze ultra-wide field images

AIM

To investigate the morphological characteristics of retinal vessels in patients with different severity of diabetic retinopathy (DR) and in patients with or without diabetic macular edema (DME).

METHODS

The 239 eyes of DR patients and 100 eyes of healthy individuals were recruited for the study. The severity of DR patients was graded as mild, moderate and severe non-proliferative diabetic retinopathy (NPDR) according to the international clinical diabetic retinopathy (ICDR) disease severity scale classification, and retinal vascular morphology was quantitatively analyzed in ultra-wide field images using RU-net and transfer learning methods. The presence of DME was determined by optical coherence tomography (OCT), and differences in vascular morphological characteristics were compared between patients with and without DME.

RESULTS

Retinal vessel segmentation using RU-net and transfer learning system had an accuracy of 99% and a Dice metric of 0.76. Compared with the healthy group, the DR group had smaller vessel angles (33.68±3.01 vs 37.78±1.60), smaller fractal dimension (Df) values (1.33±0.05 vs 1.41±0.03), less vessel density (1.12±0.44 vs 2.09±0.36) and fewer vascular branches (206.1±88.8 vs 396.5±91.3), all P<0.001. As the severity of DR increased, Df values decreased, P=0.031. No significant difference between the DME and non-DME groups were observed in vascular morphological characteristics.

CONCLUSION

In this study, an artificial intelligence retinal vessel segmentation system is used with 99% accuracy, thus providing with relatively satisfactory performance in the evaluation of quantitative vascular morphology. DR patients have a tendency of vascular occlusion and dropout. The presence of DME does not compromise the integral retinal vascular pattern.

A Swin transformer encoder-based StyleGAN for unbalanced endoscopic image enhancement

With the rapid development of artificial intelligence, automated endoscopy-assisted diagnostic systems have become an effective tool for reducing the diagnostic costs and shortening the treatment cycle of patients. Typically, the performance of these systems depends on deep learning models which are pre-trained with large-scale labeled data, for example, early gastric cancer based on endoscopic images. However, the expensive annotation and the subjectivity of the annotators lead to an insufficient and class-imbalanced endoscopic image dataset, and these datasets are detrimental to the training of deep learning models. Therefore, we proposed a Swin Transformer encoder-based StyleGAN (STE-StyleGAN) for unbalanced endoscopic image enhancement, which is composed of an adversarial learning encoder and generator. Firstly, a pre-trained Swin Transformer is introduced into the encoder to extract multi-scale features layer by layer from endoscopic images. The features are subsequently fed into a mapping block for aggregation and recombination. Secondly, a self-attention mechanism is applied to the generator, which adds detailed information of the image layer by layer through recoded features, enabling the generator to autonomously learn the coupling between different image regions. Finally, we conducted extensive experiments on a private intestinal metaplasia grading dataset from a Grade-A tertiary hospital. The experimental results show that the images generated by STE-StyleGAN are closer to the initial image distribution, achieving a Fréchet Inception Distance (FID) value of 100.4. Then, these generated images are used to enhance the initial dataset to improve the robustness of the classification model, and achieved a top accuracy of 86 %.

Optical coherence tomography choroidal enhancement using generative deep learning

Spectral-domain optical coherence tomography (SDOCT) is the gold standard of imaging the eye in clinics. Penetration depth with such devices is, however, limited and visualization of the choroid, which is essential for diagnosing chorioretinal disease, remains limited. Whereas swept-source OCT (SSOCT) devices allow for visualization of the choroid these instruments are expensive and availability in praxis is limited. We present an artificial intelligence (AI)-based solution to enhance the visualization of the choroid in OCT scans and allow for quantitative measurements of choroidal metrics using generative deep learning (DL). Synthetically enhanced SDOCT B-scans with improved choroidal visibility were generated, leveraging matching images to learn deep anatomical features during the training. Using a single-center tertiary eye care institution cohort comprising a total of 362 SDOCT-SSOCT paired subjects, we trained our model with 150,784 images from 410 healthy, 192 glaucoma, and 133 diabetic retinopathy eyes. An independent external test dataset of 37,376 images from 146 eyes was deployed to assess the authenticity and quality of the synthetically enhanced SDOCT images. Experts' ability to differentiate real versus synthetic images was poor (47.5% accuracy). Measurements of choroidal thickness, area, volume, and vascularity index, from the reference SSOCT and synthetically enhanced SDOCT, showed high Pearson's correlations of 0.97 [95% CI: 0.96-0.98], 0.97 [0.95-0.98], 0.95 [0.92-0.98], and 0.87 [0.83-0.91], with intra-class correlation values of 0.99 [0.98-0.99], 0.98 [0.98-0.99], and 0.95 [0.96-0.98], 0.93 [0.91-0.95], respectively. Thus, our DL generative model successfully generated realistic enhanced SDOCT data that is indistinguishable from SSOCT images providing improved visualization of the choroid. This technology enabled accurate measurements of choroidal metrics previously limited by the imaging depth constraints of SDOCT. The findings open new possibilities for utilizing affordable SDOCT devices in studying the choroid in both healthy and pathological conditions.

Automatic detection and differential diagnosis of age-related macular degeneration from color fundus photographs using deep learning with hierarchical vision transformer

Age-related macular degeneration (AMD) is a leading cause of vision loss in the elderly, highlighting the need for early and accurate detection. In this study, we proposed DeepDrAMD, a hierarchical vision transformer-based deep learning model that integrates data augmentation techniques and SwinTransformer, to detect AMD and distinguish between different subtypes using color fundus photographs (CFPs). The DeepDrAMD was trained on the in-house WMUEH training set and achieved high performance in AMD detection with an AUC of 98.76% in the WMUEH testing set and 96.47% in the independent external Ichallenge-AMD cohort. Furthermore, the DeepDrAMD effectively classified dryAMD and wetAMD, achieving AUCs of 93.46% and 91.55%, respectively, in the WMUEH cohort and another independent external ODIR cohort. Notably, DeepDrAMD excelled at distinguishing between wetAMD subtypes, achieving an AUC of 99.36% in the WMUEH cohort. Comparative analysis revealed that the DeepDrAMD outperformed conventional deep-learning models and expert-level diagnosis. The cost-benefit analysis demonstrated that the DeepDrAMD offers substantial cost savings and efficiency improvements compared to manual reading approaches. Overall, the DeepDrAMD represents a significant advancement in AMD detection and differential diagnosis using CFPs, and has the potential to assist healthcare professionals in informed decision-making, early intervention, and treatment optimization.

Reducing Ophthalmic Health Disparities Through Transfer Learning: A Novel Application to Overcome Data Inequality

Purpose

Race disparities in the healthcare system and the resulting inequality in clinical data among different races hinder the ability to generate equitable prediction results. This study aims to reduce healthcare disparities arising from data imbalance by leveraging advanced transfer learning (TL) methods.

Method

We examined the ophthalmic healthcare disparities at a population level using electronic medical records data from a study cohort (N = 785) receiving care at an academic institute. Regression-based TL models were usesd, transferring valuable information from the dominant racial group (White) to improve visual field mean deviation (MD) rate of change prediction particularly for data-disadvantaged African American (AA) and Asian racial groups. Prediction results of TL models were compared with two conventional approaches.

Results

Disparities in socioeconomic status and baseline disease severity were observed among the AA and Asian racial groups. The TL approach achieved marked to comparable improvement in prediction accuracy compared to the two conventional approaches as evident by smaller mean absolute errors or mean square errors. TL identified distinct key features of visual field MD rate of change for each racial group.

Conclusions

The study introduces a novel application of TL that improved reliability of the analysis in comparison with conventional methods, especially in small sample size groups. This can improve assessment of healthcare disparity and subsequent remedy approach.

Translational Relevance

TL offers an equitable and efficient approach to mitigate healthcare disparities analysis by enhancing prediction performance for data-disadvantaged group.

Deep Learning Performance of Ultra-Widefield Fundus Imaging for Screening Retinal Lesions in Rural Locales

Importance

Retinal diseases are the leading cause of irreversible blindness worldwide, and timely detection contributes to prevention of permanent vision loss, especially for patients in rural areas with limited medical resources. Deep learning systems (DLSs) based on fundus images with a 45° field of view have been extensively applied in population screening, while the feasibility of using ultra-widefield (UWF) fundus image-based DLSs to detect retinal lesions in patients in rural areas warrants exploration.

Objective

To explore the performance of a DLS for multiple retinal lesion screening using UWF fundus images from patients in rural areas.

Design, Setting, and Participants

In this diagnostic study, a previously developed DLS based on UWF fundus images was used to screen for 5 retinal lesions (retinal exudates or drusen, glaucomatous optic neuropathy, retinal hemorrhage, lattice degeneration or retinal breaks, and retinal detachment) in 24 villages of Yangxi County, China, between November 17, 2020, and March 30, 2021.

Interventions

The captured images were analyzed by the DLS and ophthalmologists.

Main Outcomes and Measures

The performance of the DLS in rural screening was compared with that of the internal validation in the previous model development stage. The image quality, lesion proportion, and complexity of lesion composition were compared between the model development stage and the rural screening stage.

Results

A total of 6222 eyes in 3149 participants (1685 women [53.5%]; mean [SD] age, 70.9 [9.1] years) were screened. The DLS achieved a mean (SD) area under the receiver operating characteristic curve (AUC) of 0.918 (0.021) (95% CI, 0.892-0.944) for detecting 5 retinal lesions in the entire data set when applied for patients in rural areas, which was lower than that reported at the model development stage (AUC, 0.998 [0.002] [95% CI, 0.995-1.000]; P < .001). Compared with the fundus images in the model development stage, the fundus images in this rural screening study had an increased frequency of poor quality (13.8% [860 of 6222] vs 0%), increased variation in lesion proportions (0.1% [6 of 6222]-36.5% [2271 of 6222] vs 14.0% [2793 of 19 891]-21.3% [3433 of 16 138]), and an increased complexity of lesion composition.

Conclusions and Relevance

This diagnostic study suggests that the DLS exhibited excellent performance using UWF fundus images as a screening tool for 5 retinal lesions in patients in a rural setting. However, poor image quality, diverse lesion proportions, and a complex set of lesions may have reduced the performance of the DLS; these factors in targeted screening scenarios should be taken into consideration in the model development stage to ensure good performance.

Improving OCT Image Segmentation of Retinal Layers by Utilizing a Machine Learning Based Multistage System of Stacked Multiscale Encoders and Decoders

Optical coherence tomography (OCT)-based retinal imagery is often utilized to determine influential factors in patient progression and treatment, for which the retinal layers of the human eye are investigated to assess a patient's health status and eyesight. In this contribution, we propose a machine learning (ML)-based multistage system of stacked multiscale encoders and decoders for the image segmentation of OCT imagery of the retinal layers to enable the following evaluation regarding the physiological and pathological states. Our proposed system's results highlight its benefits compared to currently investigated approaches by combining commonly deployed methods from deep learning (DL) while utilizing deep neural networks (DNN). We conclude that by stacking multiple multiscale encoders and decoders, improved scores for the image segmentation task can be achieved. Our retinal-layer-based segmentation results in a final segmentation performance of up to 82.25±0.74% for the Sørensen-Dice coefficient, outperforming the current best single-stage model by 1.55% with a score of 80.70±0.20%, given the evaluated peripapillary OCT data set. Additionally, we provide results on the data sets Duke SD-OCT, Heidelberg, and UMN to illustrate our model's performance on especially noisy data sets.

Accuracy of Artificial Intelligence in Estimating Best-Corrected Visual Acuity From Fundus Photographs in Eyes With Diabetic Macular Edema

Importance

Best-corrected visual acuity (BCVA) is a measure used to manage diabetic macular edema (DME), sometimes suggesting development of DME or consideration of initiating, repeating, withholding, or resuming treatment with anti-vascular endothelial growth factor. Using artificial intelligence (AI) to estimate BCVA from fundus images could help clinicians manage DME by reducing the personnel needed for refraction, the time presently required for assessing BCVA, or even the number of office visits if imaged remotely.

Objective

To evaluate the potential application of AI techniques for estimating BCVA from fundus photographs with and without ancillary information.

Design, Setting, and Participants

Deidentified color fundus images taken after dilation were used post hoc to train AI systems to perform regression from image to BCVA and to evaluate resultant estimation errors. Participants were patients enrolled in the VISTA randomized clinical trial through 148 weeks wherein the study eye was treated with aflibercept or laser. The data from study participants included macular images, clinical information, and BCVA scores by trained examiners following protocol refraction and VA measurement on Early Treatment Diabetic Retinopathy Study (ETDRS) charts.

Main Outcomes

Primary outcome was regression evaluated by mean absolute error (MAE); the secondary outcome included percentage of predictions within 10 letters, computed over the entire cohort as well as over subsets categorized by baseline BCVA, determined from baseline through the 148-week visit.

Results

Analysis included 7185 macular color fundus images of the study and fellow eyes from 459 participants. Overall, the mean (SD) age was 62.2 (9.8) years, and 250 (54.5%) were male. The baseline BCVA score for the study eyes ranged from 73 to 24 letters (approximate Snellen equivalent 20/40 to 20/320). Using ResNet50 architecture, the MAE for the testing set (n = 641 images) was 9.66 (95% CI, 9.05-10.28); 33% of the values (95% CI, 30%-37%) were within 0 to 5 letters and 28% (95% CI, 25%-32%) within 6 to 10 letters. For BCVA of 100 letters or less but more than 80 letters (20/10 to 20/25, n = 161) and 80 letters or less but more than 55 letters (20/32 to 20/80, n = 309), the MAE was 8.84 letters (95% CI, 7.88-9.81) and 7.91 letters (95% CI, 7.28-8.53), respectively.

Conclusions and Relevance

This investigation suggests AI can estimate BCVA directly from fundus photographs in patients with DME, without refraction or subjective visual acuity measurements, often within 1 to 2 lines on an ETDRS chart, supporting this AI concept if additional improvements in estimates can be achieved.

Synthetic artificial intelligence using generative adversarial network for retinal imaging in detection of age-related macular degeneration

Introduction

Age-related macular degeneration (AMD) is one of the leading causes of vision impairment globally and early detection is crucial to prevent vision loss. However, the screening of AMD is resource dependent and demands experienced healthcare providers. Recently, deep learning (DL) systems have shown the potential for effective detection of various eye diseases from retinal fundus images, but the development of such robust systems requires a large amount of datasets, which could be limited by prevalence of the disease and privacy of patient. As in the case of AMD, the advanced phenotype is often scarce for conducting DL analysis, which may be tackled via generating synthetic images using Generative Adversarial Networks (GANs). This study aims to develop GAN-synthesized fundus photos with AMD lesions, and to assess the realness of these images with an objective scale.

Methods

To build our GAN models, a total of 125,012 fundus photos were used from a real-world non-AMD phenotypical dataset. StyleGAN2 and human-in-the-loop (HITL) method were then applied to synthesize fundus images with AMD features. To objectively assess the quality of the synthesized images, we proposed a novel realness scale based on the frequency of the broken vessels observed in the fundus photos. Four residents conducted two rounds of gradings on 300 images to distinguish real from synthetic images, based on their subjective impression and the objective scale respectively.

Results and discussion

The introduction of HITL training increased the percentage of synthetic images with AMD lesions, despite the limited number of AMD images in the initial training dataset. Qualitatively, the synthesized images have been proven to be robust in that our residents had limited ability to distinguish real from synthetic ones, as evidenced by an overall accuracy of 0.66 (95% CI: 0.61-0.66) and Cohen's kappa of 0.320. For the non-referable AMD classes (no or early AMD), the accuracy was only 0.51. With the objective scale, the overall accuracy improved to 0.72. In conclusion, GAN models built with HITL training are capable of producing realistic-looking fundus images that could fool human experts, while our objective realness scale based on broken vessels can help identifying the synthetic fundus photos.

Counteracting Data Bias and Class Imbalance-Towards a Useful and Reliable Retinal Disease Recognition System

Multiple studies presented satisfactory performances for the treatment of various ocular diseases. To date, there has been no study that describes a multiclass model, medically accurate, and trained on large diverse dataset. No study has addressed a class imbalance problem in one giant dataset originating from multiple large diverse eye fundus image collections. To ensure a real-life clinical environment and mitigate the problem of biased medical image data, 22 publicly available datasets were merged. To secure medical validity only Diabetic Retinopathy (DR), Age-Related Macular Degeneration (AMD) and Glaucoma (GL) were included. The state-of-the-art models ConvNext, RegNet and ResNet were utilized. In the resulting dataset, there were 86,415 normal, 3787 GL, 632 AMD and 34,379 DR fundus images. ConvNextTiny achieved the best results in terms of recognizing most of the examined eye diseases with the most metrics. The overall accuracy was 80.46 ± 1.48. Specific accuracy values were: 80.01 ± 1.10 for normal eye fundus, 97.20 ± 0.66 for GL, 98.14 ± 0.31 for AMD, 80.66 ± 1.27 for DR. A suitable screening model for the most prevalent retinal diseases in ageing societies was designed. The model was developed on a diverse, combined large dataset which made the obtained results less biased and more generalizable.

Artificial intelligence and digital solutions for myopia

Myopia as an uncorrected visual impairment is recognized as a global public health issue with an increasing burden on health-care systems. Moreover, high myopia increases one's risk of developing pathologic myopia, which can lead to irreversible visual impairment. Thus, increased resources are needed for the early identification of complications, timely intervention to prevent myopia progression, and treatment of complications. Emerging artificial intelligence (AI) and digital technologies may have the potential to tackle these unmet needs through automated detection for screening and risk stratification, individualized prediction, and prognostication of myopia progression. AI applications in myopia for children and adults have been developed for the detection, diagnosis, and prediction of progression. Novel AI technologies, including multimodal AI, explainable AI, federated learning, automated machine learning, and blockchain, may further improve prediction performance, safety, accessibility, and also circumvent concerns of explainability. Digital technology advancements include digital therapeutics, self-monitoring devices, virtual reality or augmented reality technology, and wearable devices - which provide possible avenues for monitoring myopia progression and control. However, there are challenges in the implementation of these technologies, which include requirements for specific infrastructure and resources, demonstrating clinically acceptable performance and safety of data management. Nonetheless, this remains an evolving field with the potential to address the growing global burden of myopia.

Deep Learning Model for Static Ocular Torsion Detection Using Synthetically Generated Fundus Images

Purpose

The objective of the study is to develop deep learning models using synthetic fundus images to assess the direction (intorsion versus extorsion) and amount (physiologic versus pathologic) of static ocular torsion. Static ocular torsion assessment is an important clinical tool for classifying vertical ocular misalignment; however, current methods are time-intensive with steep learning curves for frontline providers.

Methods

We used a dataset (n = 276) of right eye fundus images. The disc-foveal angle was calculated using ImageJ to generate synthetic images via image rotation. Using synthetic datasets (n = 12,740 images per model) and transfer learning (the reuse of a pretrained deep learning model on a new task), we developed a binary classifier (intorsion versus extorsion) and a multiclass classifier (physiologic versus pathologic intorsion and extorsion). Model performance was evaluated on unseen synthetic and nonsynthetic data.

Results

On the synthetic dataset, the binary classifier had an accuracy and area under the receiver operating characteristic curve (AUROC) of 0.92 and 0.98, respectively, whereas the multiclass classifier had an accuracy and AUROC of 0.77 and 0.94, respectively. The binary classifier generalized well on the nonsynthetic data (accuracy = 0.94; AUROC = 1.00).

Conclusions

The direction of static ocular torsion can be detected from synthetic fundus images using deep learning methods, which is key to differentiate between vestibular misalignment (skew deviation) and ocular muscle misalignment (superior oblique palsies).

Translational Relevance

Given the robust performance of our models on real fundus images, similar strategies can be adopted for deep learning research in rare neuro-ophthalmologic diseases with limited datasets.

Deepfake Histologic Images for Enhancing Digital Pathology

A pathologist's optical microscopic examination of thinly cut, stained tissue on glass slides prepared from a formalin-fixed paraffin-embedded tissue blocks is the gold standard for tissue diagnostics. In addition, the diagnostic abilities and expertise of pathologists is dependent on their direct experience with common and rarer variant morphologies. Recently, deep learning approaches have been used to successfully show a high level of accuracy for such tasks. However, obtaining expert-level annotated images is an expensive and time-consuming task, and artificially synthesized histologic images can prove greatly beneficial. In this study, we present an approach to not only generate histologic images that reproduce the diagnostic morphologic features of common disease but also provide a user ability to generate new and rare morphologies. Our approach involves developing a generative adversarial network model that synthesizes pathology images constrained by class labels. We investigated the ability of this framework in synthesizing realistic prostate and colon tissue images and assessed the utility of these images in augmenting the diagnostic ability of machine learning methods and their usability by a panel of experienced anatomic pathologists. Synthetic data generated by our framework performed similar to real data when training a deep learning model for diagnosis. Pathologists were not able to distinguish between real and synthetic images, and their analyses showed a similar level of interobserver agreement for prostate cancer grading. We extended the approach to significantly more complex images from colon biopsies and showed that the morphology of the complex microenvironment in such tissues can be reproduced. Finally, we present the ability for a user to generate deepfake histologic images using a simple markup of sematic labels.

Automatic generation of retinal optical coherence tomography images based on generative adversarial networks

SIGNIFICANCE

The automatic generation algorithm of optical coherence tomography (OCT) images based on generative adversarial networks (GAN) can generate a large number of simulation images by a relatively small number of real images, which can effectively improve the classification performance.

AIM

We proposed an automatic generation algorithm for retinal OCT images based on GAN to alleviate the problem of insufficient images with high quality in deep learning, and put the diagnosis algorithm toward clinical application.

APPROACH

We designed a generation network based on GAN and trained the network with a data set constructed by 2014_BOE_Srinivasan and OCT2017 to acquire three models. Then, we generated a large number of images by the three models to augment age-related macular degeneration (AMD), diabetic macular edema (DME), and normal images. We evaluated the generated images by subjective visual observation, Fréchet inception distance (FID) scores, and a classification experiment.

RESULTS

Visual observation shows that the generated images have clear and similar features compared with the real images. Also, the lesion regions containing similar features in the real image and the generated image are randomly distributed in the image field of view. When the FID scores of the three types of generated images are lowest, three local optimal models are obtained for AMD, DME, and normal images, indicating the generated images have high quality and diversity. Moreover, the classification experiment results show that the model performance trained with the mixed images is better than that of the model trained with real images, in which the accuracy, sensitivity, and specificity are improved by 5.56%, 8.89%, and 2.22%. In addition, compared with the generation method based on variational auto-encoder (VAE), the method improved the accuracy, sensitivity, and specificity by 1.97%, 2.97%, and 0.99%, for the same test set.

CONCLUSIONS

The results show that our method can augment the three kinds of OCT images, not only effectively alleviating the problem of insufficient images with high quality but also improving the diagnosis performance.

Synthesizing realistic high-resolution retina image by style-based generative adversarial network and its utilization

Realistic image synthesis based on deep learning is an invaluable technique for developing high-performance computer aided diagnosis systems while protecting patient privacy. However, training a generative adversarial network (GAN) for image synthesis remains challenging because of the large amounts of data required for training various kinds of image features. This study aims to synthesize retinal images indistinguishable from real images and evaluate the efficacy of the synthesized images having a specific disease for augmenting class imbalanced datasets. The synthesized images were validated via image Turing tests, qualitative analysis by retinal specialists, and quantitative analyses on amounts and signal-to-noise ratios of vessels. The efficacy of synthesized images was verified by deep learning-based classification performance. Turing test shows that accuracy, sensitivity, and specificity of 54.0 ± 12.3%, 71.1 ± 18.8%, and 36.9 ± 25.5%, respectively. Here, sensitivity represents correctness to find real images among real datasets. Vessel amounts and average SNR comparisons show 0.43% and 1.5% difference between real and synthesized images. The classification performance after augmenting synthesized images outperforms every ratio of imbalanced real datasets. Our study shows the realistic retina images were successfully generated with insignificant differences between the real and synthesized images and shows great potential for practical applications.

k-SALSA: k-anonymous synthetic averaging of retinal images via local style alignment

The application of modern machine learning to retinal image analyses offers valuable insights into a broad range of human health conditions beyond ophthalmic diseases. Additionally, data sharing is key to fully realizing the potential of machine learning models by providing a rich and diverse collection of training data. However, the personallyidentifying nature of retinal images, encompassing the unique vascular structure of each individual, often prevents this data from being shared openly. While prior works have explored image de-identification strategies based on synthetic averaging of images in other domains (e.g. facial images), existing techniques face difficulty in preserving both privacy and clinical utility in retinal images, as we demonstrate in our work. We therefore introduce k-SALSA, a generative adversarial network (GAN)-based framework for synthesizing retinal fundus images that summarize a given private dataset while satisfying the privacy notion of k-anonymity. k-SALSA brings together state-of-the-art techniques for training and inverting GANs to achieve practical performance on retinal images. Furthermore, k-SALSA leverages a new technique, called local style alignment, to generate a synthetic average that maximizes the retention of fine-grain visual patterns in the source images, thus improving the clinical utility of the generated images. On two benchmark datasets of diabetic retinopathy (EyePACS and APTOS), we demonstrate our improvement upon existing methods with respect to image fidelity, classification performance, and mitigation of membership inference attacks. Our work represents a step toward broader sharing of retinal images for scientific collaboration. Code is available at https://github.com/hcholab/k-salsa.

Medical Staff and Resident Preferences for Using Deep Learning in Eye Disease Screening: Discrete Choice Experiment

Background

Deep learning–assisted eye disease diagnosis technology is increasingly applied in eye disease screening. However, no research has suggested the prerequisites for health care service providers and residents willing to use it.

Objective

The aim of this paper is to reveal the preferences of health care service providers and residents for using artificial intelligence (AI) in community-based eye disease screening, particularly their preference for accuracy.

Methods

Discrete choice experiments for health care providers and residents were conducted in Shanghai, China. In total, 34 medical institutions with adequate AI-assisted screening experience participated. A total of 39 medical staff and 318 residents were asked to answer the questionnaire and make a trade-off among alternative screening strategies with different attributes, including missed diagnosis rate, overdiagnosis rate, screening result feedback efficiency, level of ophthalmologist involvement, organizational form, cost, and screening result feedback form. Conditional logit models with the stepwise selection method were used to estimate the preferences.

Results

Medical staff preferred high accuracy: The specificity of deep learning models should be more than 90% (odds ratio [OR]=0.61 for 10% overdiagnosis; P<.001), which was much higher than the Food and Drug Administration standards. However, accuracy was not the residents’ preference. Rather, they preferred to have the doctors involved in the screening process. In addition, when compared with a fully manual diagnosis, AI technology was more favored by the medical staff (OR=2.08 for semiautomated AI model and OR=2.39 for fully automated AI model; P<.001), while the residents were in disfavor of the AI technology without doctors’ supervision (OR=0.24; P<.001).

Conclusions

Deep learning model under doctors’ supervision is strongly recommended, and the specificity of the model should be more than 90%. In addition, digital transformation should help medical staff move away from heavy and repetitive work and spend more time on communicating with residents.

Evaluation of Generative Adversarial Networks for High-Resolution Synthetic Image Generation of Circumpapillary Optical Coherence Tomography Images for Glaucoma

Importance

Deep learning (DL) networks require large data sets for training, which can be challenging to collect clinically. Generative models could be used to generate large numbers of synthetic optical coherence tomography (OCT) images to train such DL networks for glaucoma detection.

Objective

To assess whether generative models can synthesize circumpapillary optic nerve head OCT images of normal and glaucomatous eyes and determine the usability of synthetic images for training DL models for glaucoma detection.

Design, Setting, and Participants

Progressively growing generative adversarial network models were trained to generate circumpapillary OCT scans. Image gradeability and authenticity were evaluated on a clinical set of 100 real and 100 synthetic images by 2 clinical experts. DL networks for glaucoma detection were trained with real or synthetic images and evaluated on independent internal and external test data sets of 140 and 300 real images, respectively.

Main Outcomes and Measures

Evaluations of the clinical set between the experts were compared. Glaucoma detection performance of the DL networks was assessed using area under the curve (AUC) analysis. Class activation maps provided visualizations of the regions contributing to the respective classifications.

Results

A total of 990 normal and 862 glaucomatous eyes were analyzed. Evaluations of the clinical set were similar for gradeability (expert 1: 92.0%; expert 2: 93.0%) and authenticity (expert 1: 51.8%; expert 2: 51.3%). The best-performing DL network trained on synthetic images had AUC scores of 0.97 (95% CI, 0.95-0.99) on the internal test data set and 0.90 (95% CI, 0.87-0.93) on the external test data set, compared with AUCs of 0.96 (95% CI, 0.94-0.99) on the internal test data set and 0.84 (95% CI, 0.80-0.87) on the external test data set for the network trained with real images. An increase in the AUC for the synthetic DL network was observed with the use of larger synthetic data set sizes. Class activation maps showed that the regions of the synthetic images contributing to glaucoma detection were generally similar to that of real images.

Conclusions and Relevance

DL networks trained with synthetic OCT images for glaucoma detection were comparable with networks trained with real images. These results suggest potential use of generative models in the training of DL networks and as a means of data sharing across institutions without patient information confidentiality issues.

The Ethical and Societal Considerations for the Rise of Artificial Intelligence and Big Data in Ophthalmology

Medical specialties with access to a large amount of imaging data, such as ophthalmology, have been at the forefront of the artificial intelligence (AI) revolution in medicine, driven by deep learning (DL) and big data. With the rise of AI and big data, there has also been increasing concern on the issues of bias and privacy, which can be partially addressed by low-shot learning, generative DL, federated learning and a "model-to-data" approach, as demonstrated by various groups of investigators in ophthalmology. However, to adequately tackle the ethical and societal challenges associated with the rise of AI in ophthalmology, a more comprehensive approach is preferable. Specifically, AI should be viewed as sociotechnical, meaning this technology shapes, and is shaped by social phenomena.

Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization

This paper tackles automated categorization of Age-related Macular Degeneration (AMD), a common macular disease among people over 50. Previous research efforts mainly focus on AMD categorization with a single-modal input, let it be a color fundus photograph (CFP) or an OCT B-scan image. By contrast, we consider AMD categorization given a multi-modal input, a direction that is clinically meaningful yet mostly unexplored. Contrary to the prior art that takes a traditional approach of feature extraction plus classifier training that cannot be jointly optimized, we opt for end-to-end multi-modal Convolutional Neural Networks (MM-CNN). Our MM-CNN is instantiated by a two-stream CNN, with spatially-invariant fusion to combine information from the CFP and OCT streams. In order to visually interpret the contribution of the individual modalities to the final prediction, we extend the class activation mapping (CAM) technique to the multi-modal scenario. For effective training of MM-CNN, we develop two data augmentation methods. One is GAN-based CFP/OCT image synthesis, with our novel use of CAMs as conditional input of a high-resolution image-to-image translation GAN. The other method is Loose Pairing, which pairs a CFP image and an OCT image on the basis of their classes instead of eye identities. Experiments on a clinical dataset consisting of 1,094 CFP images and 1,289 OCT images acquired from 1,093 distinct eyes show that the proposed solution obtains better F1 and Accuracy than multiple baselines for multi-modal AMD categorization. Code and data are available at https://github.com/li-xirong/mmc-amd.

Development and Clinical Validation of Semi-Supervised Generative Adversarial Networks for Detection of Retinal Disorders in Optical Coherence Tomography Images Using Small Dataset

PURPOSE

To develop and test semi-supervised generative adversarial networks (GANs) that detect retinal disorders on optical coherence tomography (OCT) images using a small-labeled dataset.

METHODS

From a public database, we randomly chose a small supervised dataset with 400 OCT images (100 choroidal neovascularization, 100 diabetic macular edema, 100 drusen, and 100 normal) and assigned all other OCT images to unsupervised dataset (107,912 images without labeling). We adopted a semi-supervised GAN and a supervised deep learning (DL) model for automatically detecting retinal disorders from OCT images. The performance of the 2 models was compared in 3 testing datasets with different OCT devices. The evaluation metrics included accuracy, sensitivity, specificity, and the area under the receiver operating characteristic curves.

RESULTS

The local validation dataset included 1000 images with 250 from each category. The independent clinical dataset included 366 OCT images using Cirrus OCT Shanghai Shibei Hospital and 511 OCT images using RTVue OCT from Xinhua Hospital respectively. The semi-supervised GANs classifier achieved better accuracy than supervised DL model (0.91 vs 0.86 for local cell validation dataset, 0.91 vs 0.86 in the Shanghai Shibei Hospital testing dataset, and 0.93 vs 0.92 in Xinhua Hospital testing dataset). For detecting urgent referrals (choroidal neo-vascularization and diabetic macular edema) from nonurgent referrals (drusen and normal) on OCT images, the semi-supervised GANs classifier also achieved better area under the receiver operating characteristic curves than supervised DL model (0.99 vs 0.97, 0.97 vs 0.96, and 0.99 vs 0.99, respectively).

CONCLUSIONS

A semi-supervised GAN can achieve better performance than that of a supervised DL model when the labeled dataset is limited. The current study offers utility to various research and clinical studies using DL with relatively small datasets. Semi-supervised GANs can detect retinal disorders from OCT images using relatively small dataset.

From Data to Deployment: The Collaborative Community on Ophthalmic Imaging Roadmap for Artificial Intelligence in Age-Related Macular Degeneration

OBJECTIVE

Health care systems worldwide are challenged to provide adequate care for the 200 million individuals with age-related macular degeneration (AMD). Artificial intelligence (AI) has the potential to make a significant, positive impact on the diagnosis and management of patients with AMD; however, the development of effective AI devices for clinical care faces numerous considerations and challenges, a fact evidenced by a current absence of Food and Drug Administration (FDA)-approved AI devices for AMD.

PURPOSE

To delineate the state of AI for AMD, including current data, standards, achievements, and challenges.

METHODS

Members of the Collaborative Community on Ophthalmic Imaging Working Group for AI in AMD attended an inaugural meeting on September 7, 2020, to discuss the topic. Subsequently, they undertook a comprehensive review of the medical literature relevant to the topic. Members engaged in meetings and discussion through December 2021 to synthesize the information and arrive at a consensus.

RESULTS

Existing infrastructure for robust AI development for AMD includes several large, labeled data sets of color fundus photography and OCT images; however, image data often do not contain the metadata necessary for the development of reliable, valid, and generalizable models. Data sharing for AMD model development is made difficult by restrictions on data privacy and security, although potential solutions are under investigation. Computing resources may be adequate for current applications, but knowledge of machine learning development may be scarce in many clinical ophthalmology settings. Despite these challenges, researchers have produced promising AI models for AMD for screening, diagnosis, prediction, and monitoring. Future goals include defining benchmarks to facilitate regulatory authorization and subsequent clinical setting generalization.

CONCLUSIONS

Delivering an FDA-authorized, AI-based device for clinical care in AMD involves numerous considerations, including the identification of an appropriate clinical application; acquisition and development of a large, high-quality data set; development of the AI architecture; training and validation of the model; and functional interactions between the model output and clinical end user. The research efforts undertaken to date represent starting points for the medical devices that eventually will benefit providers, health care systems, and patients.

Diagnostic accuracy of current machine learning classifiers for age-related macular degeneration: a systematic review and meta-analysis

BACKGROUND AND OBJECTIVE

The objective of this study was to systematically review and meta-analyze the diagnostic accuracy of current machine learning classifiers for age-related macular degeneration (AMD). Artificial intelligence diagnostic algorithms can automatically detect and diagnose AMD through training data from large sets of fundus or OCT images. The use of AI algorithms is a powerful tool, and it is a method of obtaining a cost-effective, simple, and fast diagnosis of AMD.

METHODS

MEDLINE, EMBASE, CINAHL, and ProQuest Dissertations and Theses were searched systematically and thoroughly. Conferences held through Association for Research in Vision and Ophthalmology, American Academy of Ophthalmology, and Canadian Society of Ophthalmology were searched. Studies were screened using Covidence software and data on sensitivity, specificity and area under curve were extracted from the included studies. STATA 15.0 was used to conduct the meta-analysis.

RESULTS

Our search strategy identified 307 records from online databases and 174 records from gray literature. Total of 13 records, 64,798 subjects (and 612,429 images), were used for the quantitative analysis. The pooled estimate for sensitivity was 0.918 [95% CI: 0.678, 0.98] and specificity was 0.888 [95% CI: 0.578, 0.98] for AMD screening using machine learning classifiers. The relative odds of a positive screen test in AMD cases were 89.74 [95% CI: 3.05-2641.59] times more likely than a negative screen test in non-AMD cases. The positive likelihood ratio was 8.22 [95% CI: 1.52-44.48] and the negative likelihood ratio was 0.09 [95% CI: 0.02-0.52].

CONCLUSION

The included studies show promising results for the diagnostic accuracy of the machine learning classifiers for AMD and its implementation in clinical settings.

Novel technical and privacy-preserving technology for artificial intelligence in ophthalmology

PURPOSE OF REVIEW

The application of artificial intelligence (AI) in medicine and ophthalmology has experienced exponential breakthroughs in recent years in diagnosis, prognosis, and aiding clinical decision-making. The use of digital data has also heralded the need for privacy-preserving technology to protect patient confidentiality and to guard against threats such as adversarial attacks. Hence, this review aims to outline novel AI-based systems for ophthalmology use, privacy-preserving measures, potential challenges, and future directions of each.

RECENT FINDINGS

Several key AI algorithms used to improve disease detection and outcomes include: Data-driven, imagedriven, natural language processing (NLP)-driven, genomics-driven, and multimodality algorithms. However, deep learning systems are susceptible to adversarial attacks, and use of data for training models is associated with privacy concerns. Several data protection methods address these concerns in the form of blockchain technology, federated learning, and generative adversarial networks.

SUMMARY

AI-applications have vast potential to meet many eyecare needs, consequently reducing burden on scarce healthcare resources. A pertinent challenge would be to maintain data privacy and confidentiality while supporting AI endeavors, where data protection methods would need to rapidly evolve with AI technology needs. Ultimately, for AI to succeed in medicine and ophthalmology, a balance would need to be found between innovation and privacy.

Image collection and annotation platforms to establish a multi-source database of oral lesions

OBJECTIVE

To describe the development of a platform for image collection and annotation that resulted in a multi-sourced international image dataset of oral lesions to facilitate the development of automated lesion classification algorithms.

MATERIALS AND METHODS

We developed a web-interface, hosted on a web server to collect oral lesions images from international partners. Further, we developed a customised annotation tool, also a web-interface for systematic annotation of images to build a rich clinically labelled dataset. We evaluated the sensitivities comparing referral decisions through the annotation process with the clinical diagnosis of the lesions.

RESULTS

The image repository hosts 2474 images of oral lesions consisting of oral cancer, oral potentially malignant disorders and other oral lesions that were collected through MeMoSA^® UPLOAD. Eight-hundred images were annotated by seven oral medicine specialists on MeMoSA^® ANNOTATE, to mark the lesion and to collect clinical labels. The sensitivity in referral decision for all lesions that required a referral for cancer management/surveillance was moderate to high depending on the type of lesion (64.3%-100%).

CONCLUSION

This is the first description of a database with clinically labelled oral lesions. This database could accelerate the improvement of AI algorithms that can promote the early detection of high-risk oral lesions.

Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey

BACKGROUND

Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions.

METHODS

We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN.

RESULTS

In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns.

CONCLUSIONS

The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.

Detecting Anomalies in Retinal Diseases Using Generative, Discriminative, and Self-supervised Deep Learning

IMPORTANCE

Anomaly detectors could be pursued for retinal diagnoses based on artificial intelligence systems that may not have access to training examples for all retinal diseases in all phenotypic presentations. Possible applications could include screening of population for any retinal disease rather than a specific disease such as diabetic retinopathy, detection of novel retinal diseases or novel presentations of common retinal diseases, and detection of rare diseases with little or no data available for training.

OBJECTIVE

To study the application of anomaly detection to retinal diseases.

DESIGN, SETTING, AND PARTICIPANTS

High-resolution retinal images from the publicly available EyePACS data set with fundus images with a corresponding label ranging from 0 to 4 for representing different severities of diabetic retinopathy. Sixteen variants of anomaly detectors were designed. For evaluation, a surrogate problem was constructed, using diabetic retinopathy images, in which only retinas with nonreferable diabetic retinopathy, ie, no diabetic macular edema, and no diabetic retinopathy or mild to moderate nonproliferative diabetic retinopathy were used for training an artificial intelligence system, but both nonreferable and referable diabetic retinopathy (including diabetic macular edema or proliferative diabetic retinopathy) were used to test the system for detecting retinal disease.

MAIN OUTCOMES AND MEASURES

Anomaly detectors were evaluated by commonly accepted performance metrics, including area under the receiver operating characteristic curve, F1 score, and accuracy.

RESULTS

A total of 88 692 high-resolution retinal images of 44 346 individuals with varying severity of diabetic retinopathy were analyzed. The best performing across all anomaly detectors had an area under the receiver operating characteristic of 0.808 (95% CI, 0.789-0.827) and was obtained using an embedding method that involved a self-supervised network.

CONCLUSIONS AND RELEVANCE

This study suggests when abnormal (diseased) data, ie, referable diabetic retinopathy in this study, were not available for training of retinal diagnostic systems wherein only nonreferable diabetic retinopathy was used for training, anomaly detection techniques were useful in identifying images with and without referable diabetic retinopathy. This suggests that anomaly detectors may be used to detect retinal diseases in more generalized settings and potentially could play a role in screening of populations for retinal diseases or identifying novel diseases and phenotyping or detecting unusual presentations of common retinal diseases.

TARA: Training and Representation Alteration for AI Fairness and Domain Generalization

We propose a novel method for enforcing AI fairness with respect to protected or sensitive factors. This method uses a dual strategy performing training and representation alteration (TARA) for the mitigation of prominent causes of AI bias. It includes the use of representation learning alteration via adversarial independence to suppress the bias-inducing dependence of the data representation from protected factors and training set alteration via intelligent augmentation to address bias-causing data imbalance by using generative models that allow the fine control of sensitive factors related to underrepresented populations via domain adaptation and latent space manipulation. When testing our methods on image analytics, experiments demonstrate that TARA significantly or fully debiases baseline models while outperforming competing debiasing methods that have the same amount of information-for example, with (% overall accuracy, % accuracy gap) = (78.8, 0.5) versus the baseline method's score of (71.8, 10.5) for Eye-PACS, and (73.7, 11.8) versus (69.1, 21.7) for CelebA. Furthermore, recognizing certain limitations in current metrics used for assessing debiasing performance, we propose novel conjunctive debiasing metrics. Our experiments also demonstrate the ability of these novel metrics in assessing the Pareto efficiency of the proposed methods.

Morphological characteristics of retinal vessels in eyes with high myopia: Ultra-wide field images analyzed by artificial intelligence using a transfer learning system

Purpose

The purpose of this study is to investigate the retinal vascular morphological characteristics in high myopia patients of different severity.

Methods

317 eyes of high myopia patients and 104 eyes of healthy control subjects were included in this study. The severity of high myopia patients is classified into C0-C4 according to the Meta Analysis of the Pathologic Myopia (META-PM) classification and their vascular morphological characteristics in ultra-wide field imaging were analyzed using transfer learning methods and RU-net. Correlation with axial length (AL), best corrected visual acuity (BCVA) and age was analyzed. In addition, the vascular morphological characteristics of myopic choroidal neovascularization (mCNV) patients and their matched high myopia patients were compared.

Results

The RU-net and transfer learning system of blood vessel segmentation had an accuracy of 98.24%, a sensitivity of 71.42%, a specificity of 99.37%, a precision of 73.68% and a F1 score of 72.29. Compared with healthy control group, high myopia group had smaller vessel angle (31.12 ± 2.27 vs. 32.33 ± 2.14), smaller fractal dimension (Df) (1.383 ± 0.060 vs. 1.424 ± 0.038), smaller vessel density (2.57 ± 0.96 vs. 3.92 ± 0.93) and fewer vascular branches (201.87 ± 75.92 vs. 271.31 ± 67.37), all P < 0.001. With the increase of myopia maculopathy severity, vessel angle, Df, vessel density and vascular branches significantly decreased (all P < 0.001). There were significant correlations of these characteristics with AL, BCVA and age. Patients with mCNV tended to have larger vessel density (P < 0.001) and more vascular branches (P = 0.045).

Conclusion

The RU-net and transfer learning technology used in this study has an accuracy of 98.24%, thus has good performance in quantitative analysis of vascular morphological characteristics in Ultra-wide field images. Along with the increase of myopic maculopathy severity and the elongation of eyeball, vessel angle, Df, vessel density and vascular branches decreased. Myopic CNV patients have larger vessel density and more vascular branches.

Role of Optical Coherence Tomography Imaging in Predicting Progression of Age-Related Macular Disease: A Survey

In developed countries, age-related macular degeneration (AMD), a retinal disease, is the main cause of vision loss in the elderly. Optical Coherence Tomography (OCT) is currently the gold standard for assessing individuals for initial AMD diagnosis. In this paper, we look at how OCT imaging can be used to diagnose AMD. Our main aim is to examine and compare automated computer-aided diagnostic (CAD) systems for diagnosing and grading of AMD. We provide a brief summary, outlining the main aspects of performance assessment and providing a basis for current research in AMD diagnosis. As a result, the only viable alternative is to prevent AMD and stop both this devastating eye condition and unwanted visual impairment. On the other hand, the grading of AMD is very important in order to detect early AMD and prevent patients from reaching advanced AMD disease. In light of this, we explore the remaining issues with automated systems for AMD detection based on OCT imaging, as well as potential directions for diagnosis and monitoring systems based on OCT imaging and telemedicine applications.

Deepfakes in Ophthalmology

Purpose

Generative adversarial networks (GANs) are deep learning (DL) models that can create and modify realistic-appearing synthetic images, or deepfakes, from real images. The purpose of our study was to evaluate the ability of experts to discern synthesized retinal fundus images from real fundus images and to review the current uses and limitations of GANs in ophthalmology.

Design

Development and expert evaluation of a GAN and an informal review of the literature.

Participants

A total of 4282 image pairs of fundus images and retinal vessel maps acquired from a multicenter ROP screening program.

Methods

Pix2Pix HD, a high-resolution GAN, was first trained and validated on fundus and vessel map image pairs and subsequently used to generate 880 images from a held-out test set. Fifty synthetic images from this test set and 50 different real images were presented to 4 expert ROP ophthalmologists using a custom online system for evaluation of whether the images were real or synthetic. Literature was reviewed on PubMed and Google Scholars using combinations of the terms ophthalmology, GANs, generative adversarial networks, ophthalmology, images, deepfakes, and synthetic. Ancestor search was performed to broaden results.

Main Outcome Measures

Expert ability to discern real versus synthetic images was evaluated using percent accuracy. Statistical significance was evaluated using a Fisher exact test, with P values ≤ 0.05 thresholded for significance.

Results

The expert majority correctly identified 59% of images as being real or synthetic (P = 0.1). Experts 1 to 4 correctly identified 54%, 58%, 49%, and 61% of images (P = 0.505, 0.158, 1.000, and 0.043, respectively). These results suggest that the majority of experts could not discern between real and synthetic images. Additionally, we identified 20 implementations of GANs in the ophthalmology literature, with applications in a variety of imaging modalities and ophthalmic diseases.

Conclusions

Generative adversarial networks can create synthetic fundus images that are indiscernible from real fundus images by expert ROP ophthalmologists. Synthetic images may improve dataset augmentation for DL, may be used in trainee education, and may have implications for patient privacy.

RF-GANs: A Method to Synthesize Retinal Fundus Images Based on Generative Adversarial Network

Diabetic retinopathy (DR) is a diabetic complication affecting the eyes, which is the main cause of blindness in young and middle-aged people. In order to speed up the diagnosis of DR, a mass of deep learning methods have been used for the detection of this disease, but they failed to attain excellent results due to unbalanced training data, i.e., the lack of DR fundus images. To address the problem of data imbalance, this paper proposes a method dubbed retinal fundus images generative adversarial networks (RF-GANs), which is based on generative adversarial network, to synthesize retinal fundus images. RF-GANs is composed of two generation models, RF-GAN1 and RF-GAN2. Firstly, RF-GAN1 is employed to translate retinal fundus images from source domain (the domain of semantic segmentation datasets) to target domain (the domain of EyePACS dataset connected to Kaggle (EyePACS)). Then, we train the semantic segmentation models with the translated images, and employ the trained models to extract the structural and lesion masks (hereafter, we refer to it as Masks) of EyePACS. Finally, we employ RF-GAN2 to synthesize retinal fundus images using the Masks and DR grading labels. This paper verifies the effectiveness of the method: RF-GAN1 can narrow down the domain gap between different datasets to improve the performance of the segmentation models. RF-GAN2 can synthesize realistic retinal fundus images. Adopting the synthesized images for data augmentation, the accuracy and quadratic weighted kappa of the state-of-the-art DR grading model on the testing set of EyePACS increase by 1.53% and 1.70%, respectively.