RAC-CNN: multimodal deep learning based automatic detection and classification of rod and cone photoreceptors in adaptive optics scanning light ophthalmoscope images

Adaptive optics imaging in inherited retinal diseases: A scoping review of the clinical literature

Adaptive optics (AO) imaging enables direct, objective assessments of retinal cells. Applications of AO show great promise in advancing our understanding of the etiology of inherited retinal disease (IRDs) and discovering new imaging biomarkers. This scoping review systematically identifies and summarizes clinical studies evaluating AO imaging in IRDs. Ovid MEDLINE and EMBASE were searched on February 6, 2023. Studies describing AO imaging in monogenic IRDs were included. Study screening and data extraction were performed by 2 reviewers independently. This review presents (1) a broad overview of the dominant areas of research; (2) a summary of IRD characteristics revealed by AO imaging; and (3) a discussion of methodological considerations relating to AO imaging in IRDs. From 140 studies with AO outcomes, including 2 following subretinal gene therapy treatments, 75% included fewer than 10 participants with AO imaging data. Of 100 studies that included participants' genetic diagnoses, the most common IRD genes with AO outcomes are CNGA3, CNGB3, CHM, USH2A, and ABCA4. Confocal reflectance AO scanning laser ophthalmoscopy was the most reported imaging modality, followed by flood-illuminated AO and split-detector AO. The most common outcome was cone density, reported quantitatively in 56% of studies. Future research areas include guidelines to reduce variability in the reporting of AO methodology and a focus on functional AO techniques to guide the development of therapeutic interventions.

Deep Density Estimation for Cone Counting and Diagnosis of Genetic Eye Diseases From Adaptive Optics Scanning Light Ophthalmoscope Images

Purpose

Adaptive optics scanning light ophthalmoscope (AOSLO) imaging offers a microscopic view of the living retina, holding promise for diagnosing and researching eye diseases like retinitis pigmentosa and Stargardt's disease. The technology's clinical impact of AOSLO hinges on early detection through automated analysis tools.

Methods

We introduce Cone Density Estimation (CoDE) and CoDE for Diagnosis (CoDED). CoDE is a deep density estimation model for cone counting that estimates a density function whose integral is equal to the number of cones. CoDED is an integration of CoDE with deep image classifiers for diagnosis. We use two AOSLO image datasets to train and evaluate the performance of cone density estimation and classification models for retinitis pigmentosa and Stargardt's disease.

Results

Bland-Altman plots show that CoDE outperforms state-of-the-art models for cone density estimation. CoDED reported an F1 score of 0.770 ± 0.04 for disease classification, outperforming traditional convolutional networks.

Conclusions

CoDE shows promise in classifying the retinitis pigmentosa and Stargardt's disease cases from a single AOSLO image. Our preliminary results suggest the potential role of analyzing patterns in the retinal cellular mosaic to aid in the diagnosis of genetic eye diseases.

Translational Relevance

Our study explores the potential of deep density estimation models to aid in the analysis of AOSLO images. Although the initial results are encouraging, more research is needed to fully realize the potential of such methods in the treatment and study of genetic retinal pathologies.

Adaptive Optics Imaging of Inherited Retinal Disease

The human retina is amenable to direct, noninvasive visualization using a wide array of imaging modalities. In the ∼140 years since the publication of the first image of the living human retina, there has been a continued evolution of retinal imaging technology. Advances in image acquisition and processing speed now allow real-time visualization of retinal structure, which has revolutionized the diagnosis and management of eye disease. Enormous advances have come in image resolution, with adaptive optics (AO)-based systems capable of imaging the retina with single-cell resolution. In addition, newer functional imaging techniques provide the ability to assess function with exquisite spatial and temporal resolution. These imaging advances have had an especially profound impact on the field of inherited retinal disease research. Here we will review some of the advances and applications of AO retinal imaging in patients with inherited retinal disease.

Wide-Field Three-Dimensional Depth-Invariant Cellular-Resolution Imaging of the Human Retina

Three-dimensional (3D) cellular-resolution imaging of the living human retina over a large field of view will bring a great impact in clinical ophthalmology, potentially finding new biomarkers for early diagnosis and improving the pathophysiological understanding of ocular diseases. While hardware-based and computational adaptive optics (AO) optical coherence tomography (OCT) have been developed to achieve cellular-resolution retinal imaging, these approaches support limited 3D imaging fields, and their high cost and intrinsic hardware complexity limit their practical utility. Here, this work demonstrates 3D depth-invariant cellular-resolution imaging of the living human retina over a 3 × 3 mm field of view using the first intrinsically phase-stable multi-MHz retinal swept-source OCT and novel computational defocus and aberration correction methods. Single-acquisition imaging of photoreceptor cells, retinal nerve fiber layer, and retinal capillaries is presented across unprecedented imaging fields. By providing wide-field 3D cellular-resolution imaging in the human retina using a standard point-scan architecture routinely used in the clinic, this platform proposes a strategy for expanded utilization of high-resolution retinal imaging in both research and clinical settings.

Deep learning-enabled volumetric cone photoreceptor segmentation in adaptive optics optical coherence tomography images of normal and diseased eyes

Objective quantification of photoreceptor cell morphology, such as cell diameter and outer segment length, is crucial for early, accurate, and sensitive diagnosis and prognosis of retinal neurodegenerative diseases. Adaptive optics optical coherence tomography (AO-OCT) provides three-dimensional (3-D) visualization of photoreceptor cells in the living human eye. The current gold standard for extracting cell morphology from AO-OCT images involves the tedious process of 2-D manual marking. To automate this process and extend to 3-D analysis of the volumetric data, we propose a comprehensive deep learning framework to segment individual cone cells in AO-OCT scans. Our automated method achieved human-level performance in assessing cone photoreceptors of healthy and diseased participants captured with three different AO-OCT systems representing two different types of point scanning OCT: spectral domain and swept source.

Extracting spacing-derived estimates of rod density in healthy retinae

Quantification of the rod photoreceptor mosaic using adaptive optics scanning light ophthalmoscopy (AOSLO) remains challenging. Here we demonstrate a method for deriving estimates of rod density and rod:cone ratio based on measures of rod spacing, cone numerosity, and cone inner segment area. Twenty-two AOSLO images with complete rod visualization were used to validate this spacing-derived method for estimating density. The method was then used to estimate rod metrics in an additional 105 images without complete rod visualization. The spacing-derived rod mosaic metrics were comparable to published data from histology. This method could be leveraged to develop large normative databases of rod mosaic metrics, though limitations persist with intergrader variability in assessing cone area and numerosity.

Using deep learning for the automated identification of cone and rod photoreceptors from adaptive optics imaging of the human retina

Adaptive optics imaging has enabled the enhanced in vivo retinal visualization of individual cone and rod photoreceptors. Effective analysis of such high-resolution, feature rich images requires automated, robust algorithms. This paper describes RC-UPerNet, a novel deep learning algorithm, for identifying both types of photoreceptors, and was evaluated on images from central and peripheral retina extending out to 30° from the fovea in the nasal and temporal directions. Precision, recall and Dice scores were 0.928, 0.917 and 0.922 respectively for cones, and 0.876, 0.867 and 0.870 for rods. Scores agree well with human graders and are better than previously reported AI-based approaches.

Intergrader agreement of foveal cone topography measured using adaptive optics scanning light ophthalmoscopy

The foveal cone mosaic can be directly visualized using adaptive optics scanning light ophthalmoscopy (AOSLO). Previous studies in individuals with normal vision report wide variability in the topography of the foveal cone mosaic, especially the value of peak cone density (PCD). While these studies often involve a human grader, there have been no studies examining intergrader reproducibility of foveal cone mosaic metrics. Here we re-analyzed published AOSLO foveal cone images from 44 individuals to assess the relationship between the cone density centroid (CDC) location and the location of PCD. Across 5 graders with variable experience, we found a measurement error of 11.7% in PCD estimates and higher intergrader reproducibility of CDC location compared to PCD location (p < 0.0001). These estimates of measurement error can be used in future studies of the foveal cone mosaic, and our results support use of the CDC location as a more reproducible anchor for cross-modality analyses.

Application of Adaptive Optics in Ophthalmology

The eye, the photoreceptive organ used to perceive the external environment, is of great importance to humans. It has been proven that some diseases in humans are accompanied by fundus changes; therefore, the health status of people may be interpreted from retinal images. However, the human eye is not a perfect refractive system for the existence of ocular aberrations. These aberrations not only affect the ability of human visual discrimination and recognition, but restrict the observation of the fine structures of human eye and reduce the possibility of exploring the mechanisms of eye disease. Adaptive optics (AO) is a technique that corrects optical wavefront aberrations. Once integrated into ophthalmoscopes, AO enables retinal imaging at the cellular level. This paper illustrates the principle of AO in correcting wavefront aberrations in human eyes, and then reviews the applications and advances of AO in ophthalmology, including the adaptive optics fundus camera (AO-FC), the adaptive optics scanning laser ophthalmoscope (AO-SLO), the adaptive optics optical coherence tomography (AO-OCT), and their combined multimodal imaging technologies. The future development trend of AO in ophthalmology is also prospected.

Theoretical versus empirical measures of retinal magnification for scaling AOSLO images

The adaptive optics scanning light ophthalmoscope (AOSLO) allows cellular resolution imaging of the living retina. The accuracy of many quantitative measurements made from these images requires accurate estimates of the lateral scale of the images. Here, we used trial lenses, which are known to affect the relative magnification of the retinal image, to compare empirical measures of image scale with theoretical estimates from a four-surface optical model. The theoretical optical model overestimated the empirically determined change in image scale in 70% of the subjects examined, albeit to varying degrees. While the origin for the differences between subjects is not known, residual accommodation during imaging likely contributes to this variability in retinal magnification. These data provide an opportunity to derive improved lateral scaling error estimates for structural metrics extracted from AOSLO retinal images.

Deep learning classification of cervical dysplasia using depth-resolved angular light scattering profiles

We present a machine learning method for detecting and staging cervical dysplastic tissue using light scattering data based on a convolutional neural network (CNN) architecture. Depth-resolved angular scattering measurements from two clinical trials were used to generate independent training and validation sets as input of our model. We report 90.3% sensitivity, 85.7% specificity, and 87.5% accuracy in classifying cervical dysplasia, showing the uniformity of classification of a/LCI scans across different instruments. Further, our deep learning approach significantly improved processing speeds over the traditional Mie theory inverse light scattering analysis (ILSA) method, with a hundredfold reduction in processing time, offering a promising approach for a/LCI in the clinic for assessing cervical dysplasia.

Automated image processing pipeline for adaptive optics scanning light ophthalmoscopy

To mitigate the substantial post-processing burden associated with adaptive optics scanning light ophthalmoscopy (AOSLO), we have developed an open-source, automated AOSLO image processing pipeline with both "live" and "full" modes. The live mode provides feedback during acquisition, while the full mode is intended to automatically integrate the copious disparate modules currently used in generating analyzable montages. The mean (±SD) lag between initiation and montage placement for the live pipeline was 54.6 ± 32.7s. The full pipeline reduced overall human operator time by 54.9 ± 28.4%, with no significant difference in resultant cone density metrics. The reduced overhead decreases both the technical burden and operating cost of AOSLO imaging, increasing overall clinical accessibility.

Weakly supervised individual ganglion cell segmentation from adaptive optics OCT images for glaucomatous damage assessment

Cell-level quantitative features of retinal ganglion cells (GCs) are potentially important biomarkers for improved diagnosis and treatment monitoring of neurodegenerative diseases such as glaucoma, Parkinson's disease, and Alzheimer's disease. Yet, due to limited resolution, individual GCs cannot be visualized by commonly used ophthalmic imaging systems, including optical coherence tomography (OCT), and assessment is limited to gross layer thickness analysis. Adaptive optics OCT (AO-OCT) enables in vivo imaging of individual retinal GCs. We present an automated segmentation of GC layer (GCL) somas from AO-OCT volumes based on weakly supervised deep learning (named WeakGCSeg), which effectively utilizes weak annotations in the training process. Experimental results show that WeakGCSeg is on par with or superior to human experts and is superior to other state-of-the-art networks. The automated quantitative features of individual GCLs show an increase in structure-function correlation in glaucoma subjects compared to using thickness measures from OCT images. Our results suggest that by automatic quantification of GC morphology, WeakGCSeg can potentially alleviate a major bottleneck in using AO-OCT for vision research.

Advances in multimodal imaging in ophthalmology

Multimodality ophthalmic imaging systems aim to enhance the contrast, resolution, and functionality of existing technologies to improve disease diagnostics and therapeutic guidance. These systems include advanced acquisition and post-processing methods using optical coherence tomography (OCT), combined scanning laser ophthalmoscopy and OCT systems, adaptive optics, surgical guidance, and photoacoustic technologies. Here, we provide an overview of these ophthalmic imaging systems and their clinical and basic science applications.

Adaptive optics: principles and applications in ophthalmology

This is a comprehensive review of the principles and applications of adaptive optics (AO) in ophthalmology. It has been combined with flood illumination ophthalmoscopy, scanning laser ophthalmoscopy, as well as optical coherence tomography to image photoreceptors, retinal pigment epithelium (RPE), retinal ganglion cells, lamina cribrosa and the retinal vasculature. In this review, we highlight the clinical studies that have utilised AO to understand disease mechanisms. However, there are some limitations to using AO in a clinical setting including the cost of running an AO imaging service, the time needed to scan patients, the lack of normative databases and the very small size of area imaged. However, it is undoubtedly an exceptional research tool that enables visualisation of the retina at a cellular level.

Adaptive optics for high-resolution imaging

Adaptive optics (AO) is a technique that corrects for optical aberrations. It was originally proposed to correct for the blurring effect of atmospheric turbulence on images in ground-based telescopes and was instrumental in the work that resulted in the Nobel prize-winning discovery of a supermassive compact object at the centre of our galaxy. When AO is used to correct for the eye's imperfect optics, retinal changes at the cellular level can be detected, allowing us to study the operation of the visual system and to assess ocular health in the microscopic domain. By correcting for sample-induced blur in microscopy, AO has pushed the boundaries of imaging in thick tissue specimens, such as when observing neuronal processes in the brain. In this primer, we focus on the application of AO for high-resolution imaging in astronomy, vision science and microscopy. We begin with an overview of the general principles of AO and its main components, which include methods to measure the aberrations, devices for aberration correction, and how these components are linked in operation. We present results and applications from each field along with reproducibility considerations and limitations. Finally, we discuss future directions.

Open-Source Automatic Segmentation of Ocular Structures and Biomarkers of Microbial Keratitis on Slit-Lamp Photography Images Using Deep Learning

We propose a fully-automatic deep learning-based algorithm for segmentation of ocular structures and microbial keratitis (MK) biomarkers on slit-lamp photography (SLP) images. The dataset consisted of SLP images from 133 eyes with manual annotations by a physician, P1. A modified region-based convolutional neural network, SLIT-Net, was developed and trained using P1's annotations to identify and segment four pathological regions of interest (ROIs) on diffuse white light images (stromal infiltrate (SI), hypopyon, white blood cell (WBC) border, corneal edema border), one pathological ROI on diffuse blue light images (epithelial defect (ED)), and two non-pathological ROIs on all images (corneal limbus, light reflexes). To assess inter-reader variability, 75 eyes were manually annotated for pathological ROIs by a second physician, P2. Performance was evaluated using the Dice similarity coefficient (DSC) and Hausdorff distance (HD). Using seven-fold cross-validation, the DSC of the algorithm (as compared to P1) for all ROIs was good (range: 0.62-0.95) on all 133 eyes. For the subset of 75 eyes with manual annotations by P2, the DSC for pathological ROIs ranged from 0.69-0.85 (SLIT-Net) vs. 0.37-0.92 (P2). DSCs for SLIT-Net were not significantly different than P2 for segmenting hypopyons (p > 0.05) and higher than P2 for WBCs (p < 0.001) and edema (p < 0.001). DSCs were higher for P2 for segmenting SIs (p < 0.001) and EDs (p < 0.001). HDs were lower for P2 for segmenting SIs (p = 0.005) and EDs (p < 0.001) and not significantly different for hypopyons (p > 0.05), WBCs (p > 0.05), and edema (p > 0.05). This prototype fully-automatic algorithm to segment MK biomarkers on SLP images performed to expectations on an exploratory dataset and holds promise for quantification of corneal physiology and pathology.

Promises and pitfalls of evaluating photoreceptor-based retinal disease with adaptive optics scanning light ophthalmoscopy (AOSLO)

Adaptive optics scanning light ophthalmoscopy (AOSLO) allows visualization of the living human retina with exquisite single-cell resolution. This technology has improved our understanding of normal retinal structure and revealed pathophysiological details of a number of retinal diseases. Despite the remarkable capabilities of AOSLO, it has not seen the widespread commercial adoption and mainstream clinical success of other modalities developed in a similar time frame. Nevertheless, continued advancements in AOSLO hardware and software have expanded use to a broader range of patients. Current devices enable imaging of a number of different retinal cell types, with recent improvements in stimulus and detection schemes enabling monitoring of retinal function, microscopic structural changes, and even subcellular activity. This has positioned AOSLO for use in clinical trials, primarily as exploratory outcome measures or biomarkers that can be used to monitor disease progression or therapeutic response. AOSLO metrics could facilitate patient selection for such trials, to refine inclusion criteria or to guide the choice of therapy, depending on the presence, absence, or functional viability of specific cell types. Here we explore the potential of AOSLO retinal imaging by reviewing clinical applications as well as some of the pitfalls and barriers to more widespread clinical adoption.

Multimodal handheld adaptive optics scanning laser ophthalmoscope

Non-confocal adaptive optics scanning laser ophthalmoscopy (AOSLO) has enhanced the study of human retinal photoreceptors by providing complementary information to standard confocal AOSLO images. Previously we developed the first confocal handheld AOSLO (HAOSLO) capable of in vivo cone photoreceptor imaging in supine and non-cooperative patients. Here, we introduce the first multimodal (M-)HAOSLO for confocal and non-confocal split-detection (SD) imaging to allow for more comprehensive patient data collection. Aside from its unprecedented miniature size and weight, M-HAOSLO is also the first system to perform sensorless wavefront-corrected SD imaging of cone photoreceptors.

Cone Identification in Choroideremia: Repeatability, Reliability, and Automation Through Use of a Convolutional Neural Network

Purpose

Adaptive optics imaging has enabled the visualization of photoreceptors both in health and disease. However, there remains a need for automated accurate cone photoreceptor identification in images of disease. Here, we apply an open-source convolutional neural network (CNN) to automatically identify cones in images of choroideremia (CHM). We further compare the results to the repeatability and reliability of manual cone identifications in CHM.

Methods

We used split-detection adaptive optics scanning laser ophthalmoscopy to image the inner segment cone mosaic of 17 patients with CHM. Cones were manually identified twice by one experienced grader and once by two additional experienced graders in 204 regions of interest (ROIs). An open-source CNN either pre-trained on normal images or trained on CHM images automatically identified cones in the ROIs. True and false positive rates and Dice's coefficient were used to determine the agreement in cone locations between data sets. Interclass correlation coefficient was used to assess agreement in bound cone density.

Results

Intra- and intergrader agreement for cone density is high in CHM. CNN performance increased when it was trained on CHM images in comparison to normal, but had lower agreement than manual grading.

Conclusions

Manual cone identifications and cone density measurements are repeatable and reliable for images of CHM. CNNs show promise for automated cone selections, although additional improvements are needed to equal the accuracy of manual measurements.

Translational Relevance

These results are important for designing and interpreting longitudinal studies of cone mosaic metrics in disease progression or treatment intervention in CHM.

SPATIALLY INFORMED CNN FOR AUTOMATED CONE DETECTION IN ADAPTIVE OPTICS RETINAL IMAGES

Adaptive optics (AO) scanning laser ophthalmoscopy offers cellular level in-vivo imaging of the human cone mosaic. Existing analysis of cone photoreceptor density in AO images require accurate identification of cone cells, which is a time and labor-intensive task. Recently, several methods have been introduced for automated cone detection in AO retinal images using convolutional neural networks (CNN). However, these approaches have been limited in their ability to correctly identify cones when applied to AO images originating from different locations in the retina, due to changes to the reflectance and arrangement of the cone mosaics with eccentricity. To address these limitations, we present an adapted CNN architecture that incorporates spatial information directly into the network. Our approach, inspired by conditional generative adversarial networks, embeds the retina location from which each AO image was acquired as part of the training. Using manual cone identification as ground truth, our evaluation shows general improvement over existing approaches when detecting cones in the middle and periphery regions of the retina, but decreased performance near the fovea.

Microvasculature Segmentation and Intercapillary Area Quantification of the Deep Vascular Complex Using Transfer Learning

Purpose

Optical coherence tomography angiography (OCT-A) permits visualization of the changes to the retinal circulation due to diabetic retinopathy (DR), a microvascular complication of diabetes. We demonstrate accurate segmentation of the vascular morphology for the superficial capillary plexus (SCP) and deep vascular complex (DVC) using a convolutional neural network (CNN) for quantitative analysis.

Methods

The main CNN training dataset consisted of retinal OCT-A with a 6 × 6-mm field of view (FOV), acquired using a Zeiss PlexElite. Multiple-volume acquisition and averaging enhanced the vasculature contrast used for constructing the ground truth for neural network training. We used transfer learning from a CNN trained on smaller FOVs of the SCP acquired using different OCT instruments. Quantitative analysis of perfusion was performed on the resulting automated vasculature segmentations in representative patients with DR.

Results

The automated segmentations of the OCT-A images maintained the distinct morphologies of the SCP and DVC. The network segmented the SCP with an accuracy and Dice index of 0.8599 and 0.8618, respectively, and 0.7986 and 0.8139, respectively, for the DVC. The inter-rater comparisons for the SCP had an accuracy and Dice index of 0.8300 and 0.6700, respectively, and 0.6874 and 0.7416, respectively, for the DVC.

Conclusions

Transfer learning reduces the amount of manually annotated images required while producing high-quality automatic segmentations of the SCP and DVC that exceed inter-rater comparisons. The resulting intercapillary area quantification provides a tool for in-depth clinical analysis of retinal perfusion.

Translational Relevance

Accurate retinal microvasculature segmentation with the CNN results in improved perfusion analysis in diabetic retinopathy.