Semi-Supervised Capsule cGAN for Speckle Noise Reduction in Retinal OCT Images

Probabilistic volumetric speckle suppression in OCT using deep learning

We present a deep learning framework for volumetric speckle reduction in optical coherence tomography (OCT) based on a conditional generative adversarial network (cGAN) that leverages the volumetric nature of OCT data. In order to utilize the volumetric nature of OCT data, our network takes partial OCT volumes as input, resulting in artifact-free despeckled volumes that exhibit excellent speckle reduction and resolution preservation in all three dimensions. Furthermore, we address the ongoing challenge of generating ground truth data for supervised speckle suppression deep learning frameworks by using volumetric non-local means despeckling-TNode- to generate training data. We show that, while TNode processing is computationally demanding, it serves as a convenient, accessible gold-standard source for training data; our cGAN replicates efficient suppression of speckle while preserving tissue structures with dimensions approaching the system resolution of non-local means despeckling while being two orders of magnitude faster than TNode. We demonstrate fast, effective, and high-quality despeckling of the proposed network in different tissue types that are not part of the training. This was achieved with training data composed of just three OCT volumes and demonstrated in three different OCT systems. The open-source nature of our work facilitates re-training and deployment in any OCT system with an all-software implementation, working around the challenge of generating high-quality, speckle-free training data.

Tensor Ring Decomposition Guided Dictionary Learning for OCT Image Denoising

Optical coherence tomography (OCT) is a non-invasive and effective tool for the imaging of retinal tissue. However, the heavy speckle noise, resulting from multiple scattering of the light waves, obscures important morphological structures and impairs the clinical diagnosis of ocular diseases. In this paper, we propose a novel and powerful model known as tensor ring decomposition-guided dictionary learning (TRGDL) for OCT image denoising, which can simultaneously utilize two useful complementary priors, i.e., three-dimensional low-rank and sparsity priors, under a unified framework. Specifically, to effectively use the strong correlation between nearby OCT frames, we construct the OCT group tensors by extracting cubic patches from OCT images and clustering similar patches. Then, since each created OCT group tensor has a low-rank structure, to exploit spatial, non-local, and its temporal correlations in a balanced way, we enforce the TR decomposition model on each OCT group tensor. Next, to use the beneficial three-dimensional inter-group sparsity, we learn shared dictionaries in both spatial and temporal dimensions from all of the stacked OCT group tensors. Furthermore, we develop an effective algorithm to solve the resulting optimization problem by using two efficient optimization approaches, including proximal alternating minimization and the alternative direction method of multipliers. Finally, extensive experiments on OCT datasets from various imaging devices are conducted to prove the generality and usefulness of the proposed TRGDL model. Experimental simulation results show that the suggested TRGDL model outperforms state-of-the-art approaches for OCT image denoising both qualitatively and quantitatively.

Total variation regularized tensor ring decomposition for OCT image denoising and super-resolution

This paper suggests a novel hybrid tensor-ring (TR) decomposition and first-order tensor-based total variation (FOTTV) model, known as the TRFOTTV model, for super-resolution and noise suppression of optical coherence tomography (OCT) images. OCT imaging faces two fundamental problems undermining correct OCT-based diagnosis: significant noise levels and low sampling rates to speed up the capturing process. Inspired by the effectiveness of TR decomposition in analyzing complicated data structures, we suggest the TRFOTTV model for noise suppression and super-resolution of OCT images. Initially, we extract the nonlocal 3D patches from OCT data and group them to create a third-order low-rank tensor. Subsequently, using TR decomposition, we extract the correlations among all modes of the grouped OCT tensor. Finally, FOTTV is integrated into the TR model to enhance spatial smoothness in OCT images and conserve layer structures more effectively. The proximal alternating minimization and alternative direction method of multipliers are applied to solve the obtained optimization problem. The effectiveness of the suggested method is verified by four OCT datasets, demonstrating superior visual and numerical outcomes compared to state-of-the-art procedures.

Toward Ground-Truth Optical Coherence Tomography via Three-Dimensional Unsupervised Deep Learning Processing and Data

Optical coherence tomography (OCT) can perform non-invasive high-resolution three-dimensional (3D) imaging and has been widely used in biomedical fields, while it is inevitably affected by coherence speckle noise which degrades OCT imaging performance and restricts its applications. Here we present a novel speckle-free OCT imaging strategy, named toward-ground-truth OCT ( t GT-OCT), that utilizes unsupervised 3D deep-learning processing and leverages OCT 3D imaging features to achieve speckle-free OCT imaging. Specifically, our proposed t GT-OCT utilizes an unsupervised 3D-convolution deep-learning network trained using random 3D volumetric data to distinguish and separate speckle from real structures in 3D imaging volumetric space; moreover, t GT-OCT effectively further reduces speckle noise and reveals structures that would otherwise be obscured by speckle noise while preserving spatial resolution. Results derived from different samples demonstrated the high-quality speckle-free 3D imaging performance of t GT-OCT and its advancement beyond the previous state-of-the-art. The code is available online: https://github.com/Voluntino/tGT-OCT.

Denoising of Optical Coherence Tomography Images in Ophthalmology Using Deep Learning: A Systematic Review

Imaging from optical coherence tomography (OCT) is widely used for detecting retinal diseases, localization of intra-retinal boundaries, etc. It is, however, degraded by speckle noise. Deep learning models can aid with denoising, allowing clinicians to clearly diagnose retinal diseases. Deep learning models can be considered as an end-to-end framework. We selected denoising studies that used deep learning models with retinal OCT imagery. Each study was quality-assessed through image quality metrics (including the peak signal-to-noise ratio-PSNR, contrast-to-noise ratio-CNR, and structural similarity index metric-SSIM). Meta-analysis could not be performed due to heterogeneity in the methods of the studies and measurements of their performance. Multiple databases (including Medline via PubMed, Google Scholar, Scopus, Embase) and a repository (ArXiv) were screened for publications published after 2010, without any limitation on language. From the 95 potential studies identified, a total of 41 were evaluated thoroughly. Fifty-four of these studies were excluded after full text assessment depending on whether deep learning (DL) was utilized or the dataset and results were not effectively explained. Numerous types of OCT images are mentioned in this review consisting of public retinal image datasets utilized purposefully for denoising OCT images (n = 37) and the Optic Nerve Head (ONH) (n = 4). A wide range of image quality metrics was used; PSNR and SNR that ranged between 8 and 156 dB. The minority of studies (n = 8) showed a low risk of bias in all domains. Studies utilizing ONH images produced either a PSNR or SNR value varying from 8.1 to 25.7 dB, and that of public retinal datasets was 26.4 to 158.6 dB. Further analysis on denoising models was not possible due to discrepancies in reporting that did not allow useful pooling. An increasing number of studies have investigated denoising retinal OCT images using deep learning, with a range of architectures being implemented. The reported increase in image quality metrics seems promising, while study and reporting quality are currently low.

Unsupervised OCT image despeckling with ground-truth- and repeated-scanning-free features

Optical coherence tomography (OCT) can resolve biological three-dimensional tissue structures, but it is inevitably plagued by speckle noise that degrades image quality and obscures biological structure. Recently unsupervised deep learning methods are becoming more popular in OCT despeckling but they still have to use unpaired noisy-clean images or paired noisy-noisy images. To address the above problem, we propose what we believe to be a novel unsupervised deep learning method for OCT despeckling, termed Double-free Net, which eliminates the need for ground truth data and repeated scanning by sub-sampling noisy images and synthesizing noisier images. In comparison to existing unsupervised methods, Double-free Net obtains superior denoising performance when trained on datasets comprising retinal and human tissue images without clean images. The efficacy of Double-free Net in denoising holds significant promise for diagnostic applications in retinal pathologies and enhances the accuracy of retinal layer segmentation. Results demonstrate that Double-free Net outperforms state-of-the-art methods and exhibits strong convenience and adaptability across different OCT images.

StruNet: Perceptual and low-rank regularized transformer for medical image denoising

BACKGROUND

Various types of noise artifacts inevitably exist in some medical imaging modalities due to limitations of imaging techniques, which impair either clinical diagnosis or subsequent analysis. Recently, deep learning approaches have been rapidly developed and applied on medical images for noise removal or image quality enhancement. Nevertheless, due to complexity and diversity of noise distribution representations in different medical imaging modalities, most of the existing deep learning frameworks are incapable to flexibly remove noise artifacts while retaining detailed information. As a result, it remains challenging to design an effective and unified medical image denoising method that will work across a variety of noise artifacts for different imaging modalities without requiring specialized knowledge in performing the task.

PURPOSE

In this paper, we propose a novel encoder-decoder architecture called Swin transformer-based residual u-shape Network (StruNet), for medical image denoising.

METHODS

Our StruNet adopts a well-designed block as the backbone of the encoder-decoder architecture, which integrates Swin Transformer modules with residual block in parallel connection. Swin Transformer modules could effectively learn hierarchical representations of noise artifacts via self-attention mechanism in non-overlapping shifted windows and cross-window connection, while residual block is advantageous to compensate loss of detailed information via shortcut connection. Furthermore, perceptual loss and low-rank regularization are incorporated into loss function respectively in order to constrain the denoising results on feature-level consistency and low-rank characteristics.

RESULTS

To evaluate the performance of the proposed method, we have conducted experiments on three medical imaging modalities including computed tomography (CT), optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA).

CONCLUSIONS

The results demonstrate that the proposed architecture yields a promising performance of suppressing multiform noise artifacts existing in different imaging modalities.

Graph Attention U-Net for Retinal Layer Surface Detection and Choroid Neovascularization Segmentation in OCT Images

Choroidal neovascularization (CNV) is a typical symptom of age-related macular degeneration (AMD) and is one of the leading causes for blindness. Accurate segmentation of CNV and detection of retinal layers are critical for eye disease diagnosis and monitoring. In this paper, we propose a novel graph attention U-Net (GA-UNet) for retinal layer surface detection and CNV segmentation in optical coherence tomography (OCT) images. Due to retinal layer deformation caused by CNV, it is challenging for existing models to segment CNV and detect retinal layer surfaces with the correct topological order. We propose two novel modules to address the challenge. The first module is a graph attention encoder (GAE) in a U-Net model that automatically integrates topological and pathological knowledge of retinal layers into the U-Net structure to achieve effective feature embedding. The second module is a graph decorrelation module (GDM) that takes reconstructed features by the decoder of the U-Net as inputs, it then decorrelates and removes information unrelated to retinal layer for improved retinal layer surface detection. In addition, we propose a new loss function to maintain the correct topological order of retinal layers and the continuity of their boundaries. The proposed model learns graph attention maps automatically during training and performs retinal layer surface detection and CNV segmentation simultaneously with the attention maps during inference. We evaluated the proposed model on our private AMD dataset and another public dataset. Experiment results show that the proposed model outperformed the competing methods for retinal layer surface detection and CNV segmentation and achieved new state of the arts on the datasets.

SCANet: A Unified Semi-Supervised Learning Framework for Vessel Segmentation

Automatic subcutaneous vessel imaging with near-infrared (NIR) optical apparatus can promote the accuracy of locating blood vessels, thus significantly contributing to clinical venipuncture research. Though deep learning models have achieved remarkable success in medical image segmentation, they still struggle in the subfield of subcutaneous vessel segmentation due to the scarcity and low-quality of annotated data. To relieve it, this work presents a novel semi-supervised learning framework, SCANet, that achieves accurate vessel segmentation through an alternate training strategy. The SCANet is composed of a multi-scale recurrent neural network that embeds coarse-to-fine features and two auxiliary branches, a consistency decoder and an adversarial learning branch, responsible for strengthening fine-grained details and eliminating differences between ground-truths and predictions, respectively. Equipped with a novel semi-supervised alternate training strategy, the three components work collaboratively, enabling SCANet to accurately segment vessel regions with only a handful of labeled data and abounding unlabeled data. Moreover, to mitigate the shortage of annotated data in this field, we provide a new subcutaneous vessel dataset, VESSEL-NIR. Extensive experiments on a wide variety of tasks, including the segmentation of subcutaneous vessels, retinal vessels, and skin lesions, well demonstrate the superiority and generality of our approach.

A hand-held optical coherence tomography angiography scanner based on angiography reconstruction transformer networks

Optical coherence tomography angiography (OCTA) has successfully demonstrated its viability for clinical applications in dermatology. Due to the high optical scattering property of skin, extracting high-quality OCTA images from skin tissues requires at least six-repeated scans. While the motion artifacts from the patient and the free hand-held probe can lead to a low-quality OCTA image. Our deep-learning-based scan pipeline enables fast and high-quality OCTA imaging with 0.3-s data acquisition. We utilize a fast scanning protocol with a 60 μm/pixel spatial interval rate and introduce angiography-reconstruction-transformer (ART) for 4× super-resolution of low transverse resolution OCTA images. The ART outperforms state-of-the-art networks in OCTA image super-resolution and provides a lighter network size. ART can restore microvessels while reducing the processing time by 85%, and maintaining improvements in structural similarity and peak-signal-to-noise ratio. This study represents that ART can achieve fast and flexible skin OCTA imaging while maintaining image quality.

Fast optical coherence tomography angiography image acquisition and reconstruction pipeline for skin application

Traditional high-quality OCTA images require multi-repeated scans (e.g., 4-8 repeats) in the same position, which may cause the patient to be uncomfortable. We propose a deep-learning-based pipeline that can extract high-quality OCTA images from only two-repeat OCT scans. The performance of the proposed image reconstruction U-Net (IRU-Net) outperforms the state-of-the-art UNet vision transformer and UNet in OCTA image reconstruction from a two-repeat OCT signal. The results demonstrated a mean peak-signal-to-noise ratio increased from 15.7 to 24.2; the mean structural similarity index measure improved from 0.28 to 0.59, while the OCT data acquisition time was reduced from 21 seconds to 3.5 seconds (reduced by 83%).

SNR-Net OCT: brighten and denoise low-light optical coherence tomography images via deep learning

Low-light optical coherence tomography (OCT) images generated when using low input power, low-quantum-efficiency detection units, low exposure time, or facing high-reflective surfaces, have low bright and signal-to-noise rates (SNR), and restrict OCT technique and clinical applications. While low input power, low quantum efficiency, and low exposure time can help reduce the hardware requirements and accelerate imaging speed; high-reflective surfaces are unavoidable sometimes. Here we propose a deep-learning-based technique to brighten and denoise low-light OCT images, termed SNR-Net OCT. The proposed SNR-Net OCT deeply integrated a conventional OCT setup and a residual-dense-block U-Net generative adversarial network with channel-wise attention connections trained using a customized large speckle-free SNR-enhanced brighter OCT dataset. Results demonstrated that the proposed SNR-Net OCT can brighten low-light OCT images and remove the speckle noise effectively, with enhancing SNR and maintaining the tissue microstructures well. Moreover, compared to the hardware-based techniques, the proposed SNR-Net OCT can be of lower cost and better performance.

On Machine Learning in Clinical Interpretation of Retinal Diseases Using OCT Images

Optical coherence tomography (OCT) is a noninvasive imaging technique that provides high-resolution cross-sectional retina images, enabling ophthalmologists to gather crucial information for diagnosing various retinal diseases. Despite its benefits, manual analysis of OCT images is time-consuming and heavily dependent on the personal experience of the analyst. This paper focuses on using machine learning to analyse OCT images in the clinical interpretation of retinal diseases. The complexity of understanding the biomarkers present in OCT images has been a challenge for many researchers, particularly those from nonclinical disciplines. This paper aims to provide an overview of the current state-of-the-art OCT image processing techniques, including image denoising and layer segmentation. It also highlights the potential of machine learning algorithms to automate the analysis of OCT images, reducing time consumption and improving diagnostic accuracy. Using machine learning in OCT image analysis can mitigate the limitations of manual analysis methods and provide a more reliable and objective approach to diagnosing retinal diseases. This paper will be of interest to ophthalmologists, researchers, and data scientists working in the field of retinal disease diagnosis and machine learning. By presenting the latest advancements in OCT image analysis using machine learning, this paper will contribute to the ongoing efforts to improve the diagnostic accuracy of retinal diseases.

Multi-task generative adversarial network for retinal optical coherence tomography image denoising

Objective. Optical coherence tomography (OCT) has become an essential imaging modality for the assessment of ophthalmic diseases. However, speckle noise in OCT images obscures subtle but important morphological details and hampers its clinical applications. In this work, a novel multi-task generative adversarial network (MGAN) is proposed for retinal OCT image denoising.Approach. To strengthen the preservation of retinal structural information in the OCT denoising procedure, the proposed MGAN integrates adversarial learning and multi-task learning. Specifically, the generator of MGAN simultaneously undertakes two tasks, including the denoising task and the segmentation task. The segmentation task aims at the generation of the retinal segmentation map, which can guide the denoising task to focus on the retina-related region based on the retina-attention module. In doing so, the denoising task can enhance the attention to the retinal region and subsequently protect the structural detail based on the supervision of the structural similarity index measure loss.Main results. The proposed MGAN was evaluated and analyzed on three public OCT datasets. The qualitative and quantitative comparisons show that the MGAN method can achieve higher image quality, and is more effective in both speckle noise reduction and structural information preservation than previous denoising methods.Significance. We have presented a MGAN for retinal OCT image denoising. The proposed method provides an effective way to strengthen the preservation of structural information while suppressing speckle noise, and can promote the OCT applications in the clinical observation and diagnosis of retinopathy.

Self-supervised speckle noise reduction of optical coherence tomography without clean data

Optical coherence tomography (OCT) is widely used in clinical diagnosis due to its non-invasive, real-time, and high-resolution characteristics. However, the inherent speckle noise seriously degrades the image quality, which might damage the fine structures in OCT, thus affecting the diagnosis results. In recent years, supervised deep learning-based denoising methods have shown excellent denoising ability. To train a deep denoiser, a large number of paired noisy-clean images are required, which is difficult to achieve in clinical practice, since acquiring a speckle-free OCT image requires dozens of repeated scans and image registration. In this research, we propose a self-supervised strategy that helps build a despeckling model by training it to map neighboring pixels in a single noisy OCT image. Adjacent pixel patches are randomly selected from the original OCT image to generate two similar undersampled images, which are respectively used as the input and target images for training a deep neural network. To ensure both the despeckling and the structure-preserving effects, a multi-scale pixel patch sampler and corresponding loss functions are adopted in our practice. Through quantitative evaluation and qualitative visual comparison, we found that the proposed method performs better than state-of-the-art methods regarding despeckling effects and structure preservation. Besides, the proposed method is much easier to train and deploy without the need for clean OCT images, which has great significance in clinical practice.

A review of generative adversarial network applications in optical coherence tomography image analysis

Optical coherence tomography (OCT) has revolutionized ophthalmic clinical practice and research, as a result of the high-resolution images that the method is able to capture in a fast, non-invasive manner. Although clinicians can interpret OCT images qualitatively, the ability to quantitatively and automatically analyse these images represents a key goal for eye care by providing clinicians with immediate and relevant metrics to inform best clinical practice. The range of applications and methods to analyse OCT images is rich and rapidly expanding. With the advent of deep learning methods, the field has experienced significant progress with state-of-the-art-performance for several OCT image analysis tasks. Generative adversarial networks (GANs) represent a subfield of deep learning that allows for a range of novel applications not possible in most other deep learning methods, with the potential to provide more accurate and robust analyses. In this review, the progress in this field and clinical impact are reviewed and the potential future development of applications of GANs to OCT image processing are discussed.

Application of medical imaging methods and artificial intelligence in tissue engineering and organ-on-a-chip

Organ-on-a-chip (OOC) is a new type of biochip technology. Various types of OOC systems have been developed rapidly in the past decade and found important applications in drug screening and precision medicine. However, due to the complexity in the structure of both the chip-body itself and the engineered-tissue inside, the imaging and analysis of OOC have still been a big challenge for biomedical researchers. Considering that medical imaging is moving towards higher spatial and temporal resolution and has more applications in tissue engineering, this paper aims to review medical imaging methods, including CT, micro-CT, MRI, small animal MRI, and OCT, and introduces the application of 3D printing in tissue engineering and OOC in which medical imaging plays an important role. The achievements of medical imaging assisted tissue engineering are reviewed, and the potential applications of medical imaging in organoids and OOC are discussed. Moreover, artificial intelligence - especially deep learning - has demonstrated its excellence in the analysis of medical imaging; we will also present the application of artificial intelligence in the image analysis of 3D tissues, especially for organoids developed in novel OOC systems.

Noise Reduction of OCT Images Based on the External Patch Prior Guided Internal Clustering and Morphological Analysis

Optical coherence tomography (OCT) is widely used in biomedical imaging. However, noise severely affects diagnosing and identifying diseased tissues on OCT images. Here, a noise reduction method based on the external patch prior guided internal clustering and morphological analysis (E2PGICMA) is developed to remove the noise of OCT images. The external patch prior guided internal clustering algorithm is used to reduce speckle noise. The morphological analysis algorithm is employed to the background for contrast enhancement. OCT images of in vivo normal skin tissues were analyzed to remove noise using the proposed method. The estimated standard deviations of the noise were chosen as different values for evaluating the quantitative metrics. The visual quality improvement includes more textures and fine detail preservation. The denoising effects of different methods were compared. Then, quantitative and qualitative evaluations of this proposed method were conducted. The results demonstrated that the SNR, PSNR, and XCOR were higher than those of the other noise-reduction methods, reaching 15.05 dB, 27.48 dB, and 0.9959, respectively. Furthermore, the presented method’s noise reduction ratio (NRR) reached 0.8999. This proposed method can efficiently remove the background and speckle noise. Improving the proposed noise reduction method would outperform existing state-of-the-art OCT despeckling methods.

Speckle denoising based on deep learning via a conditional generative adversarial network in digital holographic interferometry

Speckle denoising can improve digital holographic interferometry phase measurements but may affect experimental accuracy. A deep-learning-based speckle denoising algorithm is developed using a conditional generative adversarial network. Two subnetworks, namely discriminator and generator networks, which refer to the U-Net and DenseNet layer structures are used to supervise network learning quality and denoising. Datasets obtained from speckle simulations are shown to provide improved noise feature extraction. The loss function is designed by considering the peak signal-to-noise ratio parameters to improve efficiency and accuracy. The proposed method thus shows better performance than other denoising algorithms for processing experimental strain data from digital holography.

Hybrid-structure network and network comparative study for deep-learning-based speckle-modulating optical coherence tomography

Optical coherence tomography (OCT), a promising noninvasive bioimaging technique, can resolve sample three-dimensional microstructures. However, speckle noise imposes obvious limitations on OCT resolving capabilities. Here we proposed a deep-learning-based speckle-modulating OCT based on a hybrid-structure network, residual-dense-block U-Net generative adversarial network (RDBU-Net GAN), and further conducted a comprehensively comparative study to explore multi-type deep-learning architectures' abilities to extract speckle pattern characteristics and remove speckle, and resolve microstructures. This is the first time that network comparative study has been performed on a customized dataset containing mass more-general speckle patterns obtained from a custom-built speckle-modulating OCT, but not on retinal OCT datasets with limited speckle patterns. Results demonstrated that the proposed RDBU-Net GAN has a more excellent ability to extract speckle pattern characteristics and remove speckle, and resolve microstructures. This work will be useful for future studies on OCT speckle removing and deep-learning-based speckle-modulating OCT.

Novel technical and privacy-preserving technology for artificial intelligence in ophthalmology

PURPOSE OF REVIEW

The application of artificial intelligence (AI) in medicine and ophthalmology has experienced exponential breakthroughs in recent years in diagnosis, prognosis, and aiding clinical decision-making. The use of digital data has also heralded the need for privacy-preserving technology to protect patient confidentiality and to guard against threats such as adversarial attacks. Hence, this review aims to outline novel AI-based systems for ophthalmology use, privacy-preserving measures, potential challenges, and future directions of each.

RECENT FINDINGS

Several key AI algorithms used to improve disease detection and outcomes include: Data-driven, imagedriven, natural language processing (NLP)-driven, genomics-driven, and multimodality algorithms. However, deep learning systems are susceptible to adversarial attacks, and use of data for training models is associated with privacy concerns. Several data protection methods address these concerns in the form of blockchain technology, federated learning, and generative adversarial networks.

SUMMARY

AI-applications have vast potential to meet many eyecare needs, consequently reducing burden on scarce healthcare resources. A pertinent challenge would be to maintain data privacy and confidentiality while supporting AI endeavors, where data protection methods would need to rapidly evolve with AI technology needs. Ultimately, for AI to succeed in medicine and ophthalmology, a balance would need to be found between innovation and privacy.

Automatic zoning for retinopathy of prematurity with semi-supervised feature calibration adversarial learning

Retinopathy of prematurity (ROP) is an eye disease, which affects prematurely born infants with low birth weight and is one of the main causes of children's blindness globally. In recent years, there are many studies on automatic ROP diagnosis, mainly focusing on ROP screening such as "Yes/No ROP" or "Mild/Severe ROP" and presence/absence detection of "plus disease". Due to the lack of corresponding high-quality annotations, there are few studies on ROP zoning, which is one of the important indicators to evaluate the severity of ROP. Moreover, how to effectively utilize the unlabeled data to train model is also worth studying. Therefore, we propose a novel semi-supervised feature calibration adversarial learning network (SSFC-ALN) for 3-level ROP zoning, which consists of two subnetworks: a generative network and a compound network. The generative network is a U-shape network for producing the reconstructed images and its output is taken as one of the inputs of the compound network. The compound network is obtained by extending a common classification network with a discriminator, introducing adversarial mechanism into the whole training process. Because the definition of ROP tells us where and what to focus on in the fundus images, which is similar to the attention mechanism. Therefore, to further improve classification performance, a new attention mechanism based feature calibration module (FCM) is designed and embedded in the compound network. The proposed method was evaluated on 1013 fundus images of 108 patients with 3-fold cross validation strategy. Compared with other state-of-the-art classification methods, the proposed method achieves high classification performance.

Real-time OCT image denoising using a self-fusion neural network

Optical coherence tomography (OCT) has become the gold standard for ophthalmic diagnostic imaging. However, clinical OCT image-quality is highly variable and limited visualization can introduce errors in the quantitative analysis of anatomic and pathologic features-of-interest. Frame-averaging is a standard method for improving image-quality, however, frame-averaging in the presence of bulk-motion can degrade lateral resolution and prolongs total acquisition time. We recently introduced a method called self-fusion, which reduces speckle noise and enhances OCT signal-to-noise ratio (SNR) by using similarity between from adjacent frames and is more robust to motion-artifacts than frame-averaging. However, since self-fusion is based on deformable registration, it is computationally expensive. In this study a convolutional neural network was implemented to offset the computational overhead of self-fusion and perform OCT denoising in real-time. The self-fusion network was pretrained to fuse 3 frames to achieve near video-rate frame-rates. Our results showed a clear gain in peak SNR in the self-fused images over both the raw and frame-averaged OCT B-scans. This approach delivers a fast and robust OCT denoising alternative to frame-averaging without the need for repeated image acquisition. Real-time self-fusion image enhancement will enable improved localization of OCT field-of-view relative to features-of-interest and improved sensitivity for anatomic features of disease.

MsTGANet: Automatic Drusen Segmentation From Retinal OCT Images

Drusen is considered as the landmark for diagnosis of AMD and important risk factor for the development of AMD. Therefore, accurate segmentation of drusen in retinal OCT images is crucial for early diagnosis of AMD. However, drusen segmentation in retinal OCT images is still very challenging due to the large variations in size and shape of drusen, blurred boundaries, and speckle noise interference. Moreover, the lack of OCT dataset with pixel-level annotation is also a vital factor hindering the improvement of drusen segmentation accuracy. To solve these problems, a novel multi-scale transformer global attention network (MsTGANet) is proposed for drusen segmentation in retinal OCT images. In MsTGANet, which is based on U-Shape architecture, a novel multi-scale transformer non-local (MsTNL) module is designed and inserted into the top of encoder path, aiming at capturing multi-scale non-local features with long-range dependencies from different layers of encoder. Meanwhile, a novel multi-semantic global channel and spatial joint attention module (MsGCS) between encoder and decoder is proposed to guide the model to fuse different semantic features, thereby improving the model's ability to learn multi-semantic global contextual information. Furthermore, to alleviate the shortage of labeled data, we propose a novel semi-supervised version of MsTGANet (Semi-MsTGANet) based on pseudo-labeled data augmentation strategy, which can leverage a large amount of unlabeled data to further improve the segmentation performance. Finally, comprehensive experiments are conducted to evaluate the performance of the proposed MsTGANet and Semi-MsTGANet. The experimental results show that our proposed methods achieve better segmentation accuracy than other state-of-the-art CNN-based methods.

Semi-MsST-GAN: A Semi-Supervised Segmentation Method for Corneal Ulcer Segmentation in Slit-Lamp Images

Corneal ulcer is a common leading cause of corneal blindness. It is difficult to accurately segment corneal ulcers due to the following problems: large differences in the pathological shapes between point-flaky and flaky corneal ulcers, blurred boundary, noise interference, and the lack of sufficient slit-lamp images with ground truth. To address these problems, in this paper, we proposed a novel semi-supervised multi-scale self-transformer generative adversarial network (Semi-MsST-GAN) that can leverage unlabeled images to improve the performance of corneal ulcer segmentation in fluorescein staining of slit-lamp images. Firstly, to improve the performance of segmenting the corneal ulcer regions with complex pathological features, we proposed a novel multi-scale self-transformer network (MsSTNet) as the MsST-GAN generator, which can guide the model to aggregate the low-level weak semantic features with the high-level strong semantic information and adaptively learn the spatial correlation in feature maps. Then, to further improve the segmentation performance by leveraging unlabeled data, the semi-supervised approach based on the proposed MsST-GAN was explored to solve the problem of the lack of slit-lamp images with corresponding ground truth. The proposed Semi-MsST-GAN was comprehensively evaluated on the public SUSTech-SYSU dataset, which contains 354 labeled and 358 unlabeled fluorescein staining slit-lamp images. The results showed that, compared with other state-of-the-art methods, our proposed method achieves better performance with comparable efficiency.

Updates in deep learning research in ophthalmology

Ophthalmology has been one of the early adopters of artificial intelligence (AI) within the medical field. Deep learning (DL), in particular, has garnered significant attention due to the availability of large amounts of data and digitized ocular images. Currently, AI in Ophthalmology is mainly focused on improving disease classification and supporting decision-making when treating ophthalmic diseases such as diabetic retinopathy, age-related macular degeneration (AMD), glaucoma and retinopathy of prematurity (ROP). However, most of the DL systems (DLSs) developed thus far remain in the research stage and only a handful are able to achieve clinical translation. This phenomenon is due to a combination of factors including concerns over security and privacy, poor generalizability, trust and explainability issues, unfavorable end-user perceptions and uncertain economic value. Overcoming this challenge would require a combination approach. Firstly, emerging techniques such as federated learning (FL), generative adversarial networks (GANs), autonomous AI and blockchain will be playing an increasingly critical role to enhance privacy, collaboration and DLS performance. Next, compliance to reporting and regulatory guidelines, such as CONSORT-AI and STARD-AI, will be required to in order to improve transparency, minimize abuse and ensure reproducibility. Thirdly, frameworks will be required to obtain patient consent, perform ethical assessment and evaluate end-user perception. Lastly, proper health economic assessment (HEA) must be performed to provide financial visibility during the early phases of DLS development. This is necessary to manage resources prudently and guide the development of DLS.

MF-Net: Multi-Scale Information Fusion Network for CNV Segmentation in Retinal OCT Images

Choroid neovascularization (CNV) is one of the blinding ophthalmologic diseases. It is mainly caused by new blood vessels growing in choroid and penetrating Bruch's membrane. Accurate segmentation of CNV is essential for ophthalmologists to analyze the condition of the patient and specify treatment plan. Although many deep learning-based methods have achieved promising results in many medical image segmentation tasks, CNV segmentation in retinal optical coherence tomography (OCT) images is still very challenging as the blur boundary of CNV, large morphological differences, speckle noise, and other similar diseases interference. In addition, the lack of pixel-level annotation data is also one of the factors that affect the further improvement of CNV segmentation accuracy. To improve the accuracy of CNV segmentation, a novel multi-scale information fusion network (MF-Net) based on U-Shape architecture is proposed for CNV segmentation in retinal OCT images. A novel multi-scale adaptive-aware deformation module (MAD) is designed and inserted into the top of the encoder path, aiming at guiding the model to focus on multi-scale deformation of the targets, and aggregates the contextual information. Meanwhile, to improve the ability of the network to learn to supplement low-level local high-resolution semantic information to high-level feature maps, a novel semantics-details aggregation module (SDA) between encoder and decoder is proposed. In addition, to leverage unlabeled data to further improve the CNV segmentation, a semi-supervised version of MF-Net is designed based on pseudo-label data augmentation strategy, which can leverage unlabeled data to further improve CNV segmentation accuracy. Finally, comprehensive experiments are conducted to validate the performance of the proposed MF-Net and SemiMF-Net. The experiment results show that both proposed MF-Net and SemiMF-Net outperforms other state-of-the-art algorithms.