Tutorial on Biostatistics: Receiver-Operating Characteristic (ROC) Analysis for Correlated Eye Data

Predicting Glaucoma Surgical Outcomes Using Neural Networks and Machine Learning on Electronic Health Records

Purpose

To develop machine learning (ML) and deep learning (DL) models to predict glaucoma surgical outcomes, including postoperative intraocular pressure, use of ocular antihypertensive medications, and need for repeat surgery.

Methods

We identified glaucoma surgeries performed at Stanford from 2013-2024, with two or more postoperative visits with intraocular pressure (IOP) measurement. Patient features were identified from the electronic health record (EHR), including demographics, prior diagnosis and procedure codes, medications and eye exam findings. Classical ML and DL models were developed to predict which glaucoma surgeries would result in surgical failure, defined as (1) IOP not reduced by more than 20% of preoperative baseline on two consecutive postoperative visits, (2) increased classes of glaucoma medications, and (3) need for additional glaucoma surgery or revision of original surgery.

Results

A total of 2398 glaucoma surgeries of 1571 patients were included, of which 1677 surgeries met failure criteria. Random forest performed best for prediction of overall surgical failure, with accuracy of 75.5% and area under the receiver operator curve (AUROC) of 76.7%, similar to the deep learning model (accuracy 75.5%, AUROC 76.6%). Across all models, prediction performance was better for IOP outcomes (AUROC 86%) than need for an additional surgery (AUROC 76%) or need for additional glaucoma medication (AUC 70%).

Conclusions

ML and DL algorithms can predict glaucoma surgery outcomes using structured data inputs from EHRs.

Translational Relevance

Models that predict outcomes of glaucoma surgery may one day provide the basis for clinical decision support tools supporting surgeons in personalizing glaucoma treatment plans.

Values of macular ganglion cell-inner plexiform layer and 10-2 visual field measurements in detecting and evaluating glaucoma

AIM

To assess the performance of macular ganglion cell-inner plexiform layer thickness (mGCIPLT) and 10-2 visual field (VF) parameters in detecting early glaucoma and evaluating the severity of advanced glaucoma.

METHODS

Totally 127 eyes from 89 participants (36 eyes of 19 healthy participants, 45 eyes of 31 early glaucoma patients and 46 eyes of 39 advanced glaucoma patients) were included. The relationships between the optical coherence tomography (OCT)-derived parameters and VF sensitivity were determined. Patients with early glaucoma were divided into eyes with or without central 10° of the VF damages (CVFDs), and the diagnostic performances of OCT-derived parameters were assessed.

RESULTS

In early glaucoma, the mGCIPLT was significantly correlated with 10-2 VF pattern standard deviation (PSD; with average mGCIPLT: β=-0.046, 95%CI, -0.067 to -0.024, P<0.001). In advanced glaucoma, the mGCIPLT was related to the 24-2 VF mean deviation (MD; with average mGCIPLT: β=0.397, 95%CI, 0.199 to 0.595, P<0.001), 10-2 VF MD (with average mGCIPLT: β=0.762, 95%CI, 0.485 to 1.038, P<0.001) and 24-2 VF PSD (with average mGCIPLT: β=0.244, 95%CI, 0.124 to 0.364, P<0.001). Except for the minimum and superotemporal mGCIPLT, the decrease of mGCIPLT in early glaucomatous eyes with CVFDs was more severe than that of early glaucomatous eyes without CVFDs. The area under the curve (AUC) of the average mGCIPLT (AUC=0.949, 95%CI, 0.868 to 0.982) was greater than that of the average circumpapillary retinal nerve fiber layer thickness (cpRNFLT; AUC=0.827, 95%CI, 0.674 to 0.918) and rim area (AUC=0.799, 95%CI, 0.610 to 0.907) in early glaucomatous eyes with CVFDs versus normal eyes.

CONCLUSION

The 10-2 VF and mGCIPLT parameters are complementary to 24-2 VF, cpRNFLT and ONH parameters, especially in detecting early glaucoma with CVFDs and evaluating the severity of advanced glaucoma in group level.

Baseline characteristics of eyes with early residual fluid post loading phase of aflibercept therapy in neovascular AMD: PRECISE study report 3

PURPOSE

To compare the baseline characteristics in patients with and without early residual fluid (ERF) after aflibercept loading phase (LP) in patients with treatment naïve neovascular age related macular degeneration (nAMD).

METHODS

Patients with nAMD initiated on LP of three intravitreal aflibercept doses were recruited from December 2019 to August 2021. Baseline demographic and OCT features associated with any ERF were analysed using Generalised Estimating Equations to account for inter-eye correlation. Receiver operating characteristic (ROC) curve was performed for selection of CST threshold.

RESULTS

Of 2128 patients enrolled, 1999 eyes of 1862 patients with complete data were included. After LP, ERF was present in 1000 (50.0%), eSRF in 746(37.3%) and eIRF in 428 (21.4%) eyes. In multivariable analysis of baseline features, eyes with increased central subfield thickness (CST) (OR 1.31 per 100 microns increase [95% CI 1.22 to 1.41]; P < 0.001), eyes with IRF and SRF at baseline (1.62 [95% CI 1.17 to 2.22]; P = 0.003), and those with SRF only (OR 2.26 [95% CI 1.59 to 3.20]; P < 0.001) relative to IRF only were determinants of ERF. CST ≥ 418 microns had 57% sensitivity and 58% specificity to distinguish ERF from no ERF at visit 4.

CONCLUSION

On average, 50% of eyes have ERF after aflibercept LP. Clinically relevant baseline determinants of ERF include CST ≥ 418 µ and presence of only SRF. These eyes may require further monthly treatment before extending treatment intervals.

Identifying Individuals at High Risk for HIV and Sexually Transmitted Infections With an Artificial Intelligence-Based Risk Assessment Tool

Background

We have previously developed an artificial intelligence-based risk assessment tool to identify the individual risk of HIV and sexually transmitted infections (STIs) in a sexual health clinical setting. Based on this tool, this study aims to determine the optimal risk score thresholds to identify individuals at high risk for HIV/STIs.

Methods

Using 2008-2022 data from 216 252 HIV, 227 995 syphilis, 262 599 gonorrhea, and 320 355 chlamydia consultations at a sexual health center, we applied MySTIRisk machine learning models to estimate infection risk scores. Optimal cutoffs for determining high-risk individuals were determined using Youden's index.

Results

The HIV risk score cutoff for high risk was 0.56, with 86.0% sensitivity (95% CI, 82.9%-88.7%) and 65.6% specificity (95% CI, 65.4%-65.8%). Thirty-five percent of participants were classified as high risk, which accounted for 86% of HIV cases. The corresponding cutoffs were 0.49 for syphilis (sensitivity, 77.6%; 95% CI, 76.2%-78.9%; specificity, 78.1%; 95% CI, 77.9%-78.3%), 0.52 for gonorrhea (sensitivity, 78.3%; 95% CI, 77.6%-78.9%; specificity, 71.9%; 95% CI, 71.7%-72.0%), and 0.47 for chlamydia (sensitivity, 68.8%; 95% CI, 68.3%-69.4%; specificity, 63.7%; 95% CI, 63.5%-63.8%). High-risk groups identified using these thresholds accounted for 78% of syphilis, 78% of gonorrhea, and 69% of chlamydia cases. The odds of positivity were significantly higher in the high-risk group than otherwise across all infections: 11.4 (95% CI, 9.3-14.8) times for HIV, 12.3 (95% CI, 11.4-13.3) for syphilis, 9.2 (95% CI, 8.8-9.6) for gonorrhea, and 3.9 (95% CI, 3.8-4.0) for chlamydia.

Conclusions

Risk scores generated by the AI-based risk assessment tool MySTIRisk, together with Youden's index, are effective in determining high-risk subgroups for HIV/STIs. The thresholds can aid targeted HIV/STI screening and prevention.

Development and international validation of custom-engineered and code-free deep-learning models for detection of plus disease in retinopathy of prematurity: a retrospective study

BACKGROUND

Retinopathy of prematurity (ROP), a leading cause of childhood blindness, is diagnosed through interval screening by paediatric ophthalmologists. However, improved survival of premature neonates coupled with a scarcity of available experts has raised concerns about the sustainability of this approach. We aimed to develop bespoke and code-free deep learning-based classifiers for plus disease, a hallmark of ROP, in an ethnically diverse population in London, UK, and externally validate them in ethnically, geographically, and socioeconomically diverse populations in four countries and three continents. Code-free deep learning is not reliant on the availability of expertly trained data scientists, thus being of particular potential benefit for low resource health-care settings.

METHODS

This retrospective cohort study used retinal images from 1370 neonates admitted to a neonatal unit at Homerton University Hospital NHS Foundation Trust, London, UK, between 2008 and 2018. Images were acquired using a Retcam Version 2 device (Natus Medical, Pleasanton, CA, USA) on all babies who were either born at less than 32 weeks gestational age or had a birthweight of less than 1501 g. Each images was graded by two junior ophthalmologists with disagreements adjudicated by a senior paediatric ophthalmologist. Bespoke and code-free deep learning models (CFDL) were developed for the discrimination of healthy, pre-plus disease, and plus disease. Performance was assessed internally on 200 images with the majority vote of three senior paediatric ophthalmologists as the reference standard. External validation was on 338 retinal images from four separate datasets from the USA, Brazil, and Egypt with images derived from Retcam and the 3nethra neo device (Forus Health, Bengaluru, India).

FINDINGS

Of the 7414 retinal images in the original dataset, 6141 images were used in the final development dataset. For the discrimination of healthy versus pre-plus or plus disease, the bespoke model had an area under the curve (AUC) of 0·986 (95% CI 0·973-0·996) and the CFDL model had an AUC of 0·989 (0·979-0·997) on the internal test set. Both models generalised well to external validation test sets acquired using the Retcam for discriminating healthy from pre-plus or plus disease (bespoke range was 0·975-1·000 and CFDL range was 0·969-0·995). The CFDL model was inferior to the bespoke model on discriminating pre-plus disease from healthy or plus disease in the USA dataset (CFDL 0·808 [95% CI 0·671-0·909, bespoke 0·942 [0·892-0·982]], p=0·0070). Performance also reduced when tested on the 3nethra neo imaging device (CFDL 0·865 [0·742-0·965] and bespoke 0·891 [0·783-0·977]).

INTERPRETATION

Both bespoke and CFDL models conferred similar performance to senior paediatric ophthalmologists for discriminating healthy retinal images from ones with features of pre-plus or plus disease; however, CFDL models might generalise less well when considering minority classes. Care should be taken when testing on data acquired using alternative imaging devices from that used for the development dataset. Our study justifies further validation of plus disease classifiers in ROP screening and supports a potential role for code-free approaches to help prevent blindness in vulnerable neonates.

FUNDING

National Institute for Health Research Biomedical Research Centre based at Moorfields Eye Hospital NHS Foundation Trust and the University College London Institute of Ophthalmology.

TRANSLATIONS

For the Portuguese and Arabic translations of the abstract see Supplementary Materials section.

Inner retinal layers' alterations of the microvasculature in early stages of Parkinson's disease: a cross sectional study

PURPOSE

To investigate microcirculation characteristics of the inner retinal layers at the macula and the peripapillary area using Optical Coherence Tomography Angiography (OCT-A) of patients in early stages of Parkinson's disease (PD).

METHODS

32 PD patients and 46 age- and gender-matched healthy controls were included in this cross sectional study. OCT-A imaging was performed to analyze microcirculation characteristics at each separate macular region (fovea, parafovea, and perifovea) and the peripapillary area of the inner retinal layers.

RESULTS

Individuals with PD had significantly lower parafoveal, perifoveal and total vessel density (VD) in the superficial capillary plexus (SCP) than controls (all p < 0.001), while foveal VD was higher in PD eyes than that of controls, though not statistically significant. Similarly, individuals with PD had significantly lower parafoveal, perifoveal and total perfusion in the SCP than control eyes (all p < 0.001), while foveal perfusion was significantly higher in PD eyes than that of controls (p = 0.008). PD eyes had significantly smaller FAZ area and perimeter accompanied by decreased circularity at the SCP as compared to controls (all p < 0.001). Concerning the peripapillary area, individuals with PD had significantly lower radial peripapillary capillary perfusion density and flux index at the SCP than controls (all p < 0.001). All p values remained statistically significant even after using the Bonferroni correction for multiple comparisons, except for that of foveal perfusion.

CONCLUSIONS

Our study indicates alterations of the inner retinal layers at the macula and the peripapillary area at the preliminary stages of PD. OCT-A parameters could potentially comprise imaging biomarkers for PD screening and improve the diagnostic algorithms.

Characteristics of Design and Analysis of Ophthalmic Randomized Controlled Trials: A Review of Ophthalmic Papers 2020-2021

Objective

To evaluate the recent practice of design and statistical analysis of ophthalmic randomized clinical trials (RCTs).

Design

Review of 96 ophthalmic RCTs.

Methods

Two authors (R.D., G.S.Y.) reviewed primary result papers published from January 2020 through December 2021 in Ophthalmology, JAMA Ophthalmology, American Journal of Ophthalmology, and British Journal of Ophthalmology. Data were extracted and analyzed for the characteristics of design (1-eye design, 2-eye design, paired-eye design, and subject design), sample size and power, and statistical analysis for intereye correlation adjustment, missing data, and correction for multiplicity.

Main Outcome Measures

Characteristics of trial design and statistical analysis.

Results

Among 96 RCTs, 50 (52%) used 1-eye design, 21 (22%) 2-eye design, 10 (10%) paired-eye design, and 15 (16%) subject design. In 31 trials of 2-eye design or paired-eye design, 18 (58%) trials had suboptimal analysis of data from both eyes by analyzing data from 1 eye (n = 10), taking the average of 2 eyes (n = 2), analyzing 2 eyes separately (n = 1), ignoring intereye correlation (n = 3), or not specifying how 2-eye data were analyzed (n = 2), and 13 trials (42%) properly adjusted the intereye correlation by using the mixed-effects model (n = 6), paired t test (n = 5), generalized estimating equations (n = 1), or marginal Cox regression model (n = 1). Among 96 trials, 75 (78%) provided both sample size and statistical power estimation, and 16 (17%) trials described statistical test for sample size or power estimation. Missing data in primary outcome occurred in 86 (90%) trials with a median missing data rate of 8%, 32 (37%) trials applied statistical methods for missing data, including last value carried forward (n = 10), multiple imputation (n = 14), or other approaches (n = 8). Among 25 trials with > 2 arms, 16 (64%) corrected for multiplicity using the Bonferroni procedure (n = 8), Hochberg procedure (n = 2), Gatekeeping procedure (n = 2), or hierarchical procedure (n = 4). Among 16 trials with multiple primary outcomes, 4 (25%) corrected for multiplicity by the Bonferroni procedure.

Conclusions

There are opportunities for improvement in the design and statistical analyses of ophthalmic trials, particularly in the aspects of adjustment for intereye correlation, missing data, and multiplicity. Continuing education in ophthalmology and vision research community may improve the quality of ophthalmic trials.

Financial Disclosures

Proprietary or commercial disclosure may be found after the references.

Correlation of Optical Coherence Tomography Angiography Characteristics with Visual Function to Define Vision-Threatening Diabetic Macular Ischemia

The thresholds of macular microvasculature parameters associated with mild visual impairment in diabetic macular ischemia (DMI) patients are unclear. Therefore, this prospective observational study is aimed at demonstrating the optical coherence tomography angiography parameters that best correlate with mild visual impairment (<70 Early Treatment Diabetic Retinopathy Study (ETDRS) letters, Snellen equivalent 20/40) in DMI. The study was completed at the Moorfields Eye Hospital from December 2019 to August 2021. A total of 123 eyes of 87 patients with stable-treated proliferative diabetic retinopathy following panretinal photocoagulation were recruited. DMI was defined as an irregular foveal avascular zone (FAZ) area ≥ 0.5 mm2 or a smaller FAZ area with parafoveal capillary dropout in at least one quadrant. The analysis showed that the whole image deep vascular complex vessel density (DVC VD) in the 3 × 3 mm area had the best discriminatory ability to identify participants with mild visual impairment at 41.9% (area under the curve = 0.77, sensitivity 94%, specificity 54%, likelihood ratio [LR] = 2.04), and the FAZ area had the greatest post-test LR = 4.21 at 0.64 mm2. The 3 × 3 mm whole image DVC VD and FAZ area cutoffs are useful for screening vision-threatening DMI, but DVC VD has low specificity.