Retinal optical coherence tomography image classification with label smoothing generative adversarial network

Generative artificial intelligence in ophthalmology

Generative AI has revolutionized medicine over the past several years. A generative adversarial network (GAN) is a deep learning framework that has become a powerful technique in medicine, particularly in ophthalmology and image analysis. In this paper we review the current ophthalmic literature involving GANs, and highlight key contributions in the field. We briefly touch on ChatGPT, another application of generative AI, and its potential in ophthalmology. We also explore the potential uses for GANs in ocular imaging, with a specific emphasis on 3 primary domains: image enhancement, disease identification, and generating of synthetic data. PubMed, Ovid MEDLINE, Google Scholar were searched from inception to October 30, 2022 to identify applications of GAN in ophthalmology. A total of 40 papers were included in this review. We cover various applications of GANs in ophthalmic-related imaging including optical coherence tomography, orbital magnetic resonance imaging, fundus photography, and ultrasound; however, we also highlight several challenges, that resulted in the generation of inaccurate and atypical results during certain iterations. Finally, we examine future directions and considerations for generative AI in ophthalmology.

A new computer-aided diagnosis tool based on deep learning methods for automatic detection of retinal disorders from OCT images

PURPOSE

Early detection of retinal disorders using optical coherence tomography (OCT) images can prevent vision loss. Since manual screening can be time-consuming, tedious, and fallible, we present a reliable computer-aided diagnosis (CAD) software based on deep learning. Also, we made efforts to increase the interpretability of the deep learning methods, overcome their vague and black box nature, and also understand their behavior in the diagnosis.

METHODS

We propose a novel method to improve the interpretability of the used deep neural network by embedding the rich semantic information of abnormal areas based on the ophthalmologists' interpretations and medical descriptions in the OCT images. Finally, we trained the classification network on a small subset of the online publicly available University of California San Diego (UCSD) dataset with an overall of 29,800 OCT images.

RESULTS

The experimental results on the 1000 test OCT images show that the proposed method achieves the overall precision, accuracy, sensitivity, and f1-score of 97.6%, 97.6%, 97.6%, and 97.59%, respectively. Also, the heat map images provide a clear region of interest which indicates that the interpretability of the proposed method is increased dramatically.

CONCLUSION

The proposed software can help ophthalmologists in providing a second opinion to make a decision, and primitive automated diagnoses of retinal diseases and even it can be used as a screening tool, in eye clinics. Also, the improvement of the interpretability of the proposed method causes to increase in the model generalization, and therefore, it will work properly on a wide range of other OCT datasets.

MBT: Model-Based Transformer for retinal optical coherence tomography image and video multi-classification

BACKGROUND AND OBJECTIVE

The detection of retinal diseases using optical coherence tomography (OCT) images and videos is a concrete example of a data classification problem. In recent years, Transformer architectures have been successfully applied to solve a variety of real-world classification problems. Although they have shown impressive discriminative abilities compared to other state-of-the-art models, improving their performance is essential, especially in healthcare-related problems.

METHODS

This paper presents an effective technique named model-based transformer (MBT). It is based on popular pre-trained transformer models, particularly, vision transformer, swin transformer for OCT image classification, and multiscale vision transformer for OCT video classification. The proposed approach is designed to represent OCT data by taking advantage of an approximate sparse representation technique. Then, it estimates the optimal features, and performs data classification.

RESULTS

The experiments are carried out using three real-world retinal datasets. The experimental results on OCT image and OCT video datasets show that the proposed method outperforms existing state-of-the-art deep learning approaches in terms of classification accuracy, precision, recall, and f1-score, kappa, AUC-ROC, and AUC-PR. It can also boost the performance of existing transformer models, including Vision transformer and Swin transformer for OCT image classification, and Multiscale Vision Transformers for OCT video classification.

CONCLUSIONS

This work presents an approach for the automated detection of retinal diseases. Although deep neural networks have proven great potential in ophthalmology applications, our findings demonstrate for the first time a new way to identify retinal pathologies using OCT videos instead of images. Moreover, our proposal can help researchers enhance the discriminative capacity of a variety of powerful deep learning models presented in published papers. This can be valuable for future directions in medical research and clinical practice.

Generating OCT B-Scan DME images using optimized Generative Adversarial Networks (GANs)

Diabetic Macular Edema (DME) represents a significant visual impairment among individuals with diabetes, leading to a dramatic reduction in visual acuity and potentially resulting in irreversible vision loss. Optical Coherence Tomography (OCT), a technique that produces high-resolution retinal images, plays a vital role in the clinical assessment of this condition. Physicians typically rely on OCT B-Scan images to evaluate DME severity. However, manual interpretation of these images is susceptible to errors, which can lead to detrimental consequences, such as misdiagnosis and improper treatment strategies. Hence, there is a critical need for more reliable diagnostic methods. This study aims to address this gap by proposing an automated model based on Generative Adversarial Networks (GANs) to generate OCT B-Scan images of DME. The model synthesizes images from patients' baseline OCT B-Scan images, which could potentially enhance the robustness of DME detection systems. We employ five distinct GANs in this study: Deep Convolutional GAN, Conditional GAN, CycleGAN, StyleGAN2, and StyleGAN3, drawing comparisons across their performance. Subsequently, the hyperparameters of the best-performing GAN are fine-tuned using Particle Swarm Optimization (PSO) to produce more realistic OCT images. This comparative analysis not only serves to improve the detection of DME severity using OCT images but also provides insights into the appropriate choice of GANs for the effective generation of realistic OCT images from the baseline OCT datasets.

Fuzzy Logic-Based System for Identifying the Severity of Diabetic Macular Edema from OCT B-Scan Images Using DRIL, HRF, and Cystoids

Diabetic Macular Edema (DME) is a severe ocular complication commonly found in patients with diabetes. The condition can precipitate a significant drop in VA and, in extreme cases, may result in irreversible vision loss. Optical Coherence Tomography (OCT), a technique that yields high-resolution retinal images, is often employed by clinicians to assess the extent of DME in patients. However, the manual interpretation of OCT B-scan images for DME identification and severity grading can be error-prone, with false negatives potentially resulting in serious repercussions. In this paper, we investigate an Artificial Intelligence (AI) driven system that offers an end-to-end automated model, designed to accurately determine DME severity using OCT B-Scan images. This model operates by extracting specific biomarkers such as Disorganization of Retinal Inner Layers (DRIL), Hyper Reflective Foci (HRF), and cystoids from the OCT image, which are then utilized to ascertain DME severity. The rules guiding the fuzzy logic engine are derived from contemporary research in the field of DME and its association with various biomarkers evident in the OCT image. The proposed model demonstrates high efficacy, identifying images with DRIL with 93.3% accuracy and successfully segmenting HRF and cystoids from OCT images with dice similarity coefficients of 91.30% and 95.07% respectively. This study presents a comprehensive system capable of accurately grading DME severity using OCT B-scan images, serving as a potentially invaluable tool in the clinical assessment and treatment of DME.

Evaluation of Generative Adversarial Networks for High-Resolution Synthetic Image Generation of Circumpapillary Optical Coherence Tomography Images for Glaucoma

Importance

Deep learning (DL) networks require large data sets for training, which can be challenging to collect clinically. Generative models could be used to generate large numbers of synthetic optical coherence tomography (OCT) images to train such DL networks for glaucoma detection.

Objective

To assess whether generative models can synthesize circumpapillary optic nerve head OCT images of normal and glaucomatous eyes and determine the usability of synthetic images for training DL models for glaucoma detection.

Design, Setting, and Participants

Progressively growing generative adversarial network models were trained to generate circumpapillary OCT scans. Image gradeability and authenticity were evaluated on a clinical set of 100 real and 100 synthetic images by 2 clinical experts. DL networks for glaucoma detection were trained with real or synthetic images and evaluated on independent internal and external test data sets of 140 and 300 real images, respectively.

Main Outcomes and Measures

Evaluations of the clinical set between the experts were compared. Glaucoma detection performance of the DL networks was assessed using area under the curve (AUC) analysis. Class activation maps provided visualizations of the regions contributing to the respective classifications.

Results

A total of 990 normal and 862 glaucomatous eyes were analyzed. Evaluations of the clinical set were similar for gradeability (expert 1: 92.0%; expert 2: 93.0%) and authenticity (expert 1: 51.8%; expert 2: 51.3%). The best-performing DL network trained on synthetic images had AUC scores of 0.97 (95% CI, 0.95-0.99) on the internal test data set and 0.90 (95% CI, 0.87-0.93) on the external test data set, compared with AUCs of 0.96 (95% CI, 0.94-0.99) on the internal test data set and 0.84 (95% CI, 0.80-0.87) on the external test data set for the network trained with real images. An increase in the AUC for the synthetic DL network was observed with the use of larger synthetic data set sizes. Class activation maps showed that the regions of the synthetic images contributing to glaucoma detection were generally similar to that of real images.

Conclusions and Relevance

DL networks trained with synthetic OCT images for glaucoma detection were comparable with networks trained with real images. These results suggest potential use of generative models in the training of DL networks and as a means of data sharing across institutions without patient information confidentiality issues.

HCTNet: A Hybrid ConvNet-Transformer Network for Retinal Optical Coherence Tomography Image Classification

Automatic and accurate optical coherence tomography (OCT) image classification is of great significance to computer-assisted diagnosis of retinal disease. In this study, we propose a hybrid ConvNet-Transformer network (HCTNet) and verify the feasibility of a Transformer-based method for retinal OCT image classification. The HCTNet first utilizes a low-level feature extraction module based on the residual dense block to generate low-level features for facilitating the network training. Then, two parallel branches of the Transformer and the ConvNet are designed to exploit the global and local context of the OCT images. Finally, a feature fusion module based on an adaptive re-weighting mechanism is employed to combine the extracted global and local features for predicting the category of OCT images in the testing datasets. The HCTNet combines the advantage of the convolutional neural network in extracting local features and the advantage of the vision Transformer in establishing long-range dependencies. A verification on two public retinal OCT datasets shows that our HCTNet method achieves an overall accuracy of 91.56% and 86.18%, respectively, outperforming the pure ViT and several ConvNet-based classification methods.

Construction of Computer Microscope Image Segmentation Model Based on Fourth-Order Partial Differential Equation Smoothing

In order to solve the problem of image noise, the author proposes a computer microscope image segmentation model based on the smoothing of fourth-order partial differential equations. On the basis of the functional describing the smoothness of the image by the directional curvature modulus, the author deduces a fourth-order partial differential equation (PDE) image noise reduction model, while effectively reducing noise, the edges are well preserved. The processing result of this method is a piecewise linear image, and there is a step in the gradient at the edge of the target. Taking advantage of this feature of the noise reduction results, the author proposes a new geodesic active contour model. The experimental results show that the reference method directly segments the results, iterates 10 times, and takes 160.721 seconds. Using the noise reduction model in the paper to preprocess and then using the reference method to segment the result, iterating 8 times, it takes 32.347 seconds. Conclusion. The new model is not only stable but also has strong contour extraction ability and fast convergence speed.

Design and Implementation of Obstetric Central Monitoring System Based on Medical Image Segmentation Algorithm

At present, the incidence of emergencies in obstetric care environment is gradually increasing, and different obstetric wards often have a variety of situations. Therefore, it can provide great help in clinical medicine to give early warning and plan coping plans according to different situations. This paper studied an obstetrics central surveillance system based on a medical image segmentation algorithm. Images obtained by central obstetrics monitoring are segmented, magnified in detail, and image features are extracted, collated, and trained. The normal distribution rule is used to classify the features, which are included in the feature library of the obstetric central monitoring system. In the gray space of the medical image, the statistical distribution of gray features of the medical image is described by the mixture model of Rayleigh distribution and Gaussian distribution. In the gray space of the medical image, Taylor series expansion is used to describe the linear geometric structure of medicine. The eigenvalues of Hessian matrix are introduced to obtain high-order multiscale features of medicine. The multiscale feature energy function is introduced into Markov random energy objective function to realize medical image segmentation. Compared with other segmentation algorithms, the accuracy and sensitivity of the proposed algorithm are 87.98% and 86.58%, respectively, which can clearly segment small medical features.

Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey

BACKGROUND

Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions.

METHODS

We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN.

RESULTS

In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns.

CONCLUSIONS

The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.

Multilevel Deep Feature Generation Framework for Automated Detection of Retinal Abnormalities Using OCT Images

Optical coherence tomography (OCT) images coupled with many learning techniques have been developed to diagnose retinal disorders. This work aims to develop a novel framework for extracting deep features from 18 pre-trained convolutional neural networks (CNN) and to attain high performance using OCT images. In this work, we have developed a new framework for automated detection of retinal disorders using transfer learning. This model consists of three phases: deep fused and multilevel feature extraction, using 18 pre-trained networks and tent maximal pooling, feature selection with ReliefF, and classification using the optimized classifier. The novelty of this proposed framework is the feature generation using widely used CNNs and to select the most suitable features for classification. The extracted features using our proposed intelligent feature extractor are fed to iterative ReliefF (IRF) to automatically select the best feature vector. The quadratic support vector machine (QSVM) is utilized as a classifier in this work. We have developed our model using two public OCT image datasets, and they are named database 1 (DB1) and database 2 (DB2). The proposed framework can attain 97.40% and 100% classification accuracies using the two OCT datasets, DB1 and DB2, respectively. These results illustrate the success of our model.

Glaucoma classification based on scanning laser ophthalmoscopic images using a deep learning ensemble method

This study aimed to assess the utility of optic nerve head (onh) en-face images, captured with scanning laser ophthalmoscopy (slo) during standard optical coherence tomography (oct) imaging of the posterior segment, and demonstrate the potential of deep learning (dl) ensemble method that operates in a low data regime to differentiate glaucoma patients from healthy controls. The two groups of subjects were initially categorized based on a range of clinical tests including measurements of intraocular pressure, visual fields, oct derived retinal nerve fiber layer (rnfl) thickness and dilated stereoscopic examination of onh. 227 slo images of 227 subjects (105 glaucoma patients and 122 controls) were used. A new task-specific convolutional neural network architecture was developed for slo image-based classification. To benchmark the results of the proposed method, a range of classifiers were tested including five machine learning methods to classify glaucoma based on rnfl thickness—a well-known biomarker in glaucoma diagnostics, ensemble classifier based on inception v3 architecture, and classifiers based on features extracted from the image. The study shows that cross-validation dl ensemble based on slo images achieved a good discrimination performance with up to 0.962 of balanced accuracy, outperforming all of the other tested classifiers.

Various Generative Adversarial Networks Model for Synthetic Prohibitory Sign Image Generation

A synthetic image is a critical issue for computer vision. Traffic sign images synthesized from standard models are commonly used to build computer recognition algorithms for acquiring more knowledge on various and low-cost research issues. Convolutional Neural Network (CNN) achieves excellent detection and recognition of traffic signs with sufficient annotated training data. The consistency of the entire vision system is dependent on neural networks. However, locating traffic sign datasets from most countries in the world is complicated. This work uses various generative adversarial networks (GAN) models to construct intricate images, such as Least Squares Generative Adversarial Networks (LSGAN), Deep Convolutional Generative Adversarial Networks (DCGAN), and Wasserstein Generative Adversarial Networks (WGAN). This paper also discusses, in particular, the quality of the images produced by various GANs with different parameters. For processing, we use a picture with a specific number and scale. The Structural Similarity Index (SSIM) and Mean Squared Error (MSE) will be used to measure image consistency. Between the generated image and the corresponding real image, the SSIM values will be compared. As a result, the images display a strong similarity to the real image when using more training images. LSGAN outperformed other GAN models in the experiment with maximum SSIM values achieved using 200 images as inputs, 2000 epochs, and size 32 × 32.