Training fellows for retinopathy of prematurity care: a Web-based survey

Characterization of Errors in Retinopathy of Prematurity Diagnosis by Ophthalmologists-in-Training in Middle-Income Countries

PURPOSE

To characterize common errors in the diagnosis of retinopathy of prematurity (ROP) among ophthalmologistsin-training in middle-income countries.

METHODS

In this prospective cohort study, 200 ophthalmologists-in-training from programs in Brazil, Mexico, and the Philippines participated. A secure web-based educational system was developed using a repository of more than 2,500 unique image sets of ROP, and a reference standard diagnosis was established by combining the clinical diagnosis and the image-based diagnosis by multiple experts. Twenty web-based cases of wide-field retinal images were presented, and ophthalmologists-in-training were asked to diagnose plus disease, zone, stage, and category for each eye. Trainees' responses were compared to the consensus reference standard diagnosis. Main outcome measures were frequency and types of diagnostic errors were analyzed.

RESULTS

The error rate in the diagnosis of any category of ROP was between 48% and 59% for all countries. The error rate in identifying type 2 or pre-plus disease was 77%, with a tendency for overdiagnosis (27% underdiagnosis vs 50% overdiagnosis; mean difference: 23.4; 95% CI: 12.1 to 34.7; P = .005). Misdiagnosis of treatment-requiring ROP as type 2 ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 18% with correct category diagnosis vs 69% when misdiagnosed; mean difference: 51.0; 95% CI: 49.3 to 52.7; P = .003).

CONCLUSIONS

Ophthalmologists-in-training from middle-income countries misdiagnosed ROP more than half of the time. Identification of plus disease was the salient factor leading to incorrect diagnosis. These findings emphasize the need for improved access to ROP education to improve competency in diagnosis among ophthalmologists-in-training in middle-income countries. [J Pediatr Ophthalmol Strabismus. 2023;60(5):344-352.].

Characterization of Errors in Retinopathy of Prematurity Diagnosis by Ophthalmologists-in-Training in the United States and Canada

PURPOSE

To identify the prominent factors that lead to misdiagnosis of retinopathy of prematurity (ROP) by ophthalmologists-in-training in the United States and Canada.

METHODS

This prospective cohort study included 32 ophthalmologists-in-training at six ophthalmology training programs in the United States and Canada. Twenty web-based cases of ROP using wide-field retinal images were presented, and ophthalmologists-in-training were asked to diagnose plus disease, zone, stage, and category for each eye. Responses were compared to a consensus reference standard diagnosis for accuracy, which was established by combining the clinical diagnosis and the image-based diagnosis by multiple experts. The types of diagnostic errors that occurred were analyzed with descriptive and chi-squared analysis. Main outcome measures were frequency of types (category, zone, stage, plus disease) of diagnostic errors; association of errors in zone, stage, and plus disease diagnosis with incorrectly identified category; and performance of ophthalmologists-in-training across postgraduate years.

RESULTS

Category of ROP was misdiagnosed at a rate of 48%. Errors in classification of plus disease were most commonly associated with misdiagnosis of treatment-requiring (plus error rate = 16% when treatment-requiring was correctly diagnosed vs 81% when underdiagnosed as type 2 or pre-plus; mean difference: 64.3; 95% CI: 51.9 to 76.7; P < .001) and type 2 or pre-plus (plus error rate = 35% when type 2 or pre-plus was correctly diagnosed vs 76% when overdiagnosed as treatment-requiring; mean difference: 41.0; 95% CI: 28.4 to 53.5; P < .001) disease. The diagnostic error rate of postgraduate year (PGY)-2 trainees was significantly higher than PGY-3 trainees (PGY-2 category error rate = 61% vs PGY-3 = 35%; mean difference, 25.4; 95% CI: 17.7 to 33.0; P < .001).

CONCLUSIONS

Ophthalmologists-in-training in the United States and Canada misdiagnosed ROP nearly half of the time, with incorrect identification of plus disease as a leading cause. Integration of structured learning for ROP in residency education may improve diagnostic competency. [J Pediatr Ophthalmol Strabismus. 2023;60(5):337-343.].

Operationalization of Retinopathy of Prematurity Screening by the Application of the Essential Public Health Services Framework

Retinopathy of prematurity (ROP) is one of the leading causes of preventable pediatric blindness worldwide. ROP screening programs have been previously implemented in multiple low- and middle-income countries. On a global scale, it is crucial that evidence-based, standardized screening criteria are utilized in the early detection and treatment of ROP. In this review article, we utilize the National Public Health Performance Standards (NPHPS) Ten Essential Public Health Services Model organized by the core functions of assessment, policy development, and assurance to evaluate the barriers and successes of existing ROP screening programs. This framework can be applied to countries facing the third epidemic of ROP and can be used to establish a generalized model for eye care and screening worldwide.

Assessment and management of retinopathy of prematurity in the era of anti-vascular endothelial growth factor (VEGF)

The incidence of retinopathy of prematurity (ROP) continues to rise due to the improved survival of very low birth weight infants in developed countries. This epidemic is also fueled by increased survival of preterm babies with variable use of oxygen and a lack of ROP awareness and screening services in resource-limited regions. Improvements in technology and a basic understanding of the disease pathophysiology have changed the way we screen and manage ROP, educate providers and patients, and improve ROP awareness. Advancements in imaging techniques, expansion of telemedicine services, and the potential for artificial intelligence-assisted ROP screening programs have created opportunities to improve ROP care in areas with a shortage of ophthalmologists trained in ROP. To address the gap in provider knowledge regarding ROP, the Global Education Network for Retinopathy of Prematurity (GEN-ROP) created a web-based tele-education training module that can be used to educate all providers involved in ROP, including non-physician ROP screeners. Over the past 50 years, the treatment of severe ROP has evolved from limited treatment modalities to cryotherapy and laser photocoagulation. More recently, there has been growing evidence to support the use of anti-vascular endothelial growth factor (VEGF) agents for the treatment of severe ROP. However, VEGF is known to be important in organogenesis and microvascular maintenance, and given that intravitreal anti-VEGF treatment can result in systemic VEGF suppression over a period of at least 1-12 weeks, there are concerns regarding adverse effects and long-term ocular and systemic developmental consequences of anti-VEGF therapy. Future research in ophthalmology to address the growing burden of ROP should focus on cost-effective fundus imaging devices, implementation of artificial intelligence platforms, updated treatment algorithms with optimal use of anti-VEGF and careful investigation of its long-term effects, and surgical options in advanced ROP. Addressing these unmet needs will aid the global effort against the ROP epidemic and optimize our understanding and treatment of this blinding disease.

Deep Learning for the Diagnosis of Stage in Retinopathy of Prematurity: Accuracy and Generalizability across Populations and Cameras

PURPOSE

Stage is an important feature to identify in retinal images of infants at risk of retinopathy of prematurity (ROP). The purpose of this study was to implement a convolutional neural network (CNN) for binary detection of stages 1, 2, and 3 in ROP and to evaluate its generalizability across different populations and camera systems.

DESIGN

Diagnostic validation study of CNN for stage detection.

PARTICIPANTS

Retinal fundus images obtained from preterm infants during routine ROP screenings.

METHODS

Two datasets were used: 5943 fundus images obtained by RetCam camera (Natus Medical, Pleasanton, CA) from 9 North American institutions and 5049 images obtained by 3nethra camera (Forus Health Incorporated, Bengaluru, India) from 4 hospitals in Nepal. Images were labeled based on the presence of stage by 1 to 3 expert graders. Three CNN models were trained using 5-fold cross-validation on datasets from North America alone, Nepal alone, and a combined dataset and were evaluated on 2 held-out test sets consisting of 708 and 247 images from the Nepali and North American datasets, respectively.

MAIN OUTCOME MEASURES

Convolutional neural network performance was evaluated using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), sensitivity, and specificity.

RESULTS

Both the North American- and Nepali-trained models demonstrated high performance on a test set from the same population: AUROC, 0.99; AUPRC, 0.98; sensitivity, 94%; and AUROC, 0.97; AUPRC, 0.91; and sensitivity, 73%; respectively. However, the performance of each model decreased to AUROC of 0.96 and AUPRC of 0.88 (sensitivity, 52%) and AUROC of 0.62 and AUPRC of 0.36 (sensitivity, 44%) when evaluated on a test set from the other population. Compared with the models trained on individual datasets, the model trained on a combined dataset achieved improved performance on each respective test set: sensitivity improved from 94% to 98% on the North American test set and from 73% to 82% on the Nepali test set.

CONCLUSIONS

A CNN can identify accurately the presence of ROP stage in retinal images, but performance depends on the similarity between training and testing populations. We demonstrated that internal and external performance can be improved by increasing the heterogeneity of the training dataset features of the training dataset, in this case by combining images from different populations and cameras.

Addressing the Third Epidemic of Retinopathy of Prematurity Through Telemedicine and Technology: A Systematic Review

The rising prevalence of retinopathy of prematurity (ROP) in low- and middle-income countries has increased the need for screening at-risk infants. The purpose of this article was to review the impact of tele-medicine and technology on ROP screening programs. Adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a systematic review was performed using PubMed, Pro-Quest, and Google Scholar bibliographic search engine. Terms searched included retinopathy of prematurity, telemedicine, and tele-ophthalmology. Data regarding internet access and gross domestic product per capita were obtained from the World Bank. Information was also obtained about internet access, speeds, and costs in low-income countries. There has been increasing integration of telemedicine and technology for ROP screening and management. Low-income countries are using available internet options and information and communications technology for ROP screening, which can aid in addressing the unique challenges faced by low-income countries. This provides a promising solution to the third epidemic of ROP by expanding and improving screening and management. Although telemedicine systems may serve as a cost-effective approach to facilitate delivery of health care, programs (especially in lowand middle-income countries) require national support to maintain its infrastructure. [J Pediatr Ophthalmol Strabismus. 2021;58(4):261-269.].

Development, Implementation, and Evaluation of Treatment Guidelines for Herpes Simplex Keratitis in Sydney, Australia

PURPOSE

To develop and measure the uptake of a local guideline for herpes simplex keratitis (HSK) and to standardize initial antiviral therapy in Australia.

METHODS

The Registered Nurses' Association of Ontario Toolkit: "Implementation of Best Practice Guidelines" was used to develop, implement, and evaluate the guideline at Sydney Eye Hospital. An implementation team was established to reach consensus on antiviral therapy guidelines through review of available evidence, identifying stakeholders, facilitators and barriers, creating strategies for implementation, and developing a sustainability plan. An audit of all adult HSK cases during a 6-month postguideline implementation period was conducted, and the results were compared with a preimplementation audit. A web-based survey was created to assess clinician awareness, usage, and level of knowledge of the guideline.

RESULTS

Clinicians, pharmacists, and administrative staff were identified as stakeholders. Changing clinician's behavior was the major barrier to implementation. Implementation strategies included printed and online materials and lectures to clinicians. A postimplementation audit included 85 patients, and 95 clinicians received a web-based survey. The dose of the prescribed antiviral medication was in alignment with the local guideline in 80% (51/64) of the patients compared with 73% (163/223) before implementation (P = 0.331). Stromal HSK with ulceration and keratouveitis were excluded because there were no recommendations before implementation. Over 70% of clinicians (30/41) were aware of the guideline and accessed them through educational resources.

CONCLUSIONS

Guidelines for the management of HSK may improve standardization of initial antiviral therapy in HSK. In practice, most clinicians were aware of and adhered to the local guideline.

Evaluation of real-time video from the digital indirect ophthalmoscope for telemedicine consultations in retinopathy of prematurity

Introduction

This study aimed to evaluate the validity of using real-time video indirect ophthalmoscopy for telemedicine consultations for paediatric retinal findings, using retinopathy of prematurity (ROP) as a model disease.

Methods

An ophthalmologist simultaneously performed and recorded routine ROP screenings on enrolled premature infants (aged ≤30 weeks and weighing <1500 g) using the Keeler digital indirect ophthalmoscope. Examinations were graded as no ROP, mild, type 1 or type 2. Masked to clinical findings, another ophthalmologist reviewed and graded streamed video feed transmitted at 4096 kbps. We compared the sensitivity and specificity of diagnosing ROP via streamed and stored-and-forwarded video clips to the gold-standard in-person examination.

Results

A total of 150 examinations of individual eyes from 34 babies were included. The median postmenstrual age (PMA) at delivery was 24 weeks (range 23–34 weeks), the median birth weight was 630 g (range 455–1530 g) and the median PMA at examination was 37 weeks (range 31–54 weeks). Of those infants with any ROP, the sensitivity and specificity of streamed examinations were 100% and 70.6%, respectively. For type 2 or worse ROP, the sensitivity and specificity were 92.5% and 86.1%, respectively. For type 1 ROP, the sensitivity and specificity were 100% and 99.3%, respectively. Sensitivities (unless already 100%) and specificities were slightly higher for store-and-forward evaluations.

Discussion

Streamed video feed from the digital indirect ophthalmoscope can be utilised to diagnose clinically significant ROP accurately, though store-and-forward video review yielded slightly better results.

Training of Residents and Fellows in Retinopathy of Prematurity Around the World: An International Web-Based Survey

PURPOSE

To characterize retinopathy of prematurity (ROP) training practices in international residency and fellowship programs.

METHODS

A publicly available online-based platform (http://www.SurveyMonkey.com) was used to develop a 28-question multiple-choice survey that targeted ROP screening and treatment methods. The authors solicited training programs in the Philippines, Thailand, and Taiwan.

RESULTS

Programs from three countries participated in the survey, and a total of 95 responses collected from residents, fellows, and attending ophthalmologists were analyzed. A descriptive analysis demonstrated that 45 participants (47%) reported 1% to 33% of ROP screenings were performed under direct supervision of attending ophthalmologists, and 35 (37%) reported the use of formal assessments. The majority of participants (Country A: 87%, Country B: 71%, and Country C: 75%) estimated 1% to 33% of their practice was spent screening for ROP. Notably, 44 participants (46%) reported performing zero laser photocoagulation treatments for ROP during training (Country A: 65%, Country B: 38%, and Country C: 38%).

CONCLUSIONS

International ophthalmology trainees perform a limited number of ROP examinations and laser interventions. ROP screenings are often unsupervised and lead to no formal evaluation by an attending ophthalmologist. Limited ROP training among ophthalmologists may lead to misdiagnosis and ultimately mismanagement of a patient. Loss of vision and exposure to unwarranted treatments are among the implications of such errors. The findings highlight the need to improve ROP training in international ophthalmology residency and fellowship programs. [J Pediatr Ophthalmol Strabismus. 2019;56(5):282-287.].

Systemic Solutions in Retinopathy of Prematurity

PURPOSE

To describe a framework for screening and treatment of retinopathy of prematurity (ROP) using telemedicine screening with laser and/or vascular endothelial growth factor inhibitor treatment strategies.

DESIGN

Literature review and perspective.

METHODS

Review of the literature and the author's experience.

RESULTS

Undetermined.

CONCLUSIONS

Telemedicine can be deployed on a national basis to provide quaternary-level screening for ROP. At-risk patients for treatment could then be directed to receive either laser photocoagulation or vascular endothelial growth factor inhibitor intravitreal injections followed at a later time by examination under anesthesia, fluorescein angiography, and diode laser photocoagulation as indicated.

Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks

Importance

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable.

Objective

To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs.

Design, Setting, and Participants

A deep convolutional neural network was trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017.

Exposures

A deep learning algorithm trained on retinal photographs.

Main Outcomes and Measures

Receiver operating characteristic analysis was performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts.

Results

Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre-plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93% with 94% specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts.

Conclusions and Relevance

This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

Expert Diagnosis of Plus Disease in Retinopathy of Prematurity From Computer-Based Image Analysis

IMPORTANCE

Published definitions of plus disease in retinopathy of prematurity (ROP) reference arterial tortuosity and venous dilation within the posterior pole based on a standard published photograph. One possible explanation for limited interexpert reliability for a diagnosis of plus disease is that experts deviate from the published definitions.

OBJECTIVE

To identify vascular features used by experts for diagnosis of plus disease through quantitative image analysis.

DESIGN, SETTING, AND PARTICIPANTS

A computer-based image analysis system (Imaging and Informatics in ROP [i-ROP]) was developed using a set of 77 digital fundus images, and the system was designed to classify images compared with a reference standard diagnosis (RSD). System performance was analyzed as a function of the field of view (circular crops with a radius of 1-6 disc diameters) and vessel subtype (arteries only, veins only, or all vessels). Routine ROP screening was conducted from June 29, 2011, to October 14, 2014, in neonatal intensive care units at 8 academic institutions, with a subset of 73 images independently classified by 11 ROP experts for validation. The RSD was compared with the majority diagnosis of experts.

MAIN OUTCOMES AND MEASURES

The primary outcome measure was the percentage of accuracy of the i-ROP system classification of plus disease, with the RSD as a function of the field of view and vessel type. Secondary outcome measures included the accuracy of the 11 experts compared with the RSD.

RESULTS

Accuracy of plus disease diagnosis by the i-ROP computer-based system was highest (95%; 95% CI, 94%-95%) when it incorporated vascular tortuosity from both arteries and veins and with the widest field of view (6-disc diameter radius). Accuracy was 90% or less when using only arterial tortuosity and 85% or less using a 2- to 3-disc diameter view similar to the standard published photograph. Diagnostic accuracy of the i-ROP system (95%) was comparable to that of 11 expert physicians (mean 87%, range 79%-99%).

CONCLUSIONS AND RELEVANCE

Experts in ROP appear to consider findings from beyond the posterior retina when diagnosing plus disease and consider tortuosity of both arteries and veins, in contrast with published definitions. It is feasible for a computer-based image analysis system to perform comparably with ROP experts, using manually segmented images.

Assessment of a Tele-education System to Enhance Retinopathy of Prematurity Training by International Ophthalmologists-in-Training in Mexico

PURPOSE

To evaluate a tele-education system developed to improve diagnostic competency in retinopathy of prematurity (ROP) by ophthalmologists-in-training in Mexico.

DESIGN

Prospective, randomized cohort study.

PARTICIPANTS

Fifty-eight ophthalmology residents and fellows from a training program in Mexico consented to participate. Twenty-nine of 58 trainees (50%) were randomized to the educational intervention (pretest, ROP tutorial, ROP educational chapters, and posttest), and 29 of 58 trainees (50%) were randomized to a control group (pretest and posttest only).

METHODS

A secure web-based educational system was created using clinical cases (20 pretest, 20 posttest, and 25 training chapter-based) developed from a repository of over 2500 unique image sets of ROP. For each image set used, a reference standard ROP diagnosis was established by combining the clinical diagnosis by indirect ophthalmoscope examination and image-based diagnosis by multiple experts. Trainees were presented with image-based clinical cases of ROP during a pretest, posttest, and training chapters.

MAIN OUTCOME MEASURES

The accuracy of ROP diagnosis (e.g., plus disease, zone, stage, category) was determined using sensitivity and specificity calculations from the pretest and posttest results of the educational intervention group versus control group. The unweighted kappa statistic was used to analyze the intragrader agreement for ROP diagnosis by the ophthalmologists-in-training during the pretest and posttest for both groups.

RESULTS

Trainees completing the tele-education system had statistically significant improvements (P < 0.01) in the accuracy of ROP diagnosis for plus disease, zone, stage, category, and aggressive posterior ROP (AP-ROP). Compared with the control group, trainees who completed the ROP tele-education system performed better on the posttest for accurately diagnosing plus disease (67% vs. 48%; P = 0.04) and the presence of ROP (96% vs. 91%; P < 0.01). The specificity for diagnosing AP-ROP (94% vs. 78%; P < 0.01), type 2 ROP or worse (92% vs. 84%; P = 0.04), and ROP requiring treatment (89% vs. 79%; P < 0.01) was better for the trainees completing the tele-education system compared with the control group. Intragrader agreement improved for identification of plus disease, zone, stage, and category of ROP after completion of the educational intervention.

CONCLUSIONS

A tele-education system for ROP education was effective in improving the diagnostic accuracy of ROP by ophthalmologists-in-training in Mexico. This system has the potential to increase competency in ROP diagnosis and management for ophthalmologists-in-training from middle-income nations.

Plus Disease in Retinopathy of Prematurity: Diagnostic Trends in 2016 Versus 2007

PURPOSE

To identify any temporal trends in the diagnosis of plus disease in retinopathy of prematurity (ROP) by experts.

DESIGN

Reliability analysis.

METHODS

ROP experts were recruited in 2007 and 2016 to classify 34 wide-field fundus images of ROP as plus, pre-plus, or normal, coded as "3," "2," and "1," respectively, in the database. The main outcome was the average calculated score for each image in each cohort. Secondary outcomes included correlation on the relative ordering of the images in 2016 vs 2007, interexpert agreement, and intraexpert agreement.

RESULTS

The average score for each image was higher for 30 of 34 (88%) images in 2016 compared with 2007, influenced by fewer images classified as normal (P < .01), a similar number of pre-plus (P = .52), and more classified as plus (P < .01). The mean weighted kappa values in 2006 were 0.36 (range 0.21-0.60), compared with 0.22 (range 0-0.40) in 2016. There was good correlation between rankings of disease severity between the 2 cohorts (Spearman rank correlation ρ = 0.94), indicating near-perfect agreement on relative disease severity.

CONCLUSIONS

Despite good agreement between cohorts on relative disease severity ranking, the higher average score and classifications for each image demonstrate that experts are diagnosing pre-plus and plus disease at earlier stages of disease severity in 2016, compared with 2007. This has implications for patient care, research, and teaching, and additional studies are needed to better understand this temporal trend in image-based plus disease diagnosis.

Trends in the Screening and Treatment of Retinopathy of Prematurity

OBJECTIVES

To determine the current practice patterns of retinopathy of prematurity (ROP) screening and treatment and the attitudes toward new screening and treatment modalities in level III and level IV NICUs, as reported by medical directors.

METHODS

Surveys were mailed to the medical directors of 847 level III NICUs identified in the 2011 American Academy of Pediatrics directory in April 2015. In September 2015, responses were compared with American Academy of Pediatrics guidelines and previous reports. Within-sample comparisons were made by level, setting, size, and academic status.

RESULTS

Respondents indicated that ROP screening is most often performed in their NICUs by pediatric and/or retina specialists (90%); retinal imaging devices are infrequently used (21%). Treatment is performed by pediatric (39%) and/or retina (57%) specialists in the NICU, usually under conscious sedation (60%). The most common treatment modality was laser photocoagulation (85%), followed by anti-vascular endothelial growth factor injection (20%). Some NICUs do not provide treatment services (28%), often due to a lack of ophthalmologists (78%). Respondents showed slightly more agreement (35%) than disagreement (25%) that a retinal imaging device could replace indirect ophthalmoscopy (40% were neutral). More respondents agreed than disagreed (30% vs 15%) that telemedicine for ROP screening is safe, but most were neutral (55%).

CONCLUSIONS

Screening and treatment of ROP are not implemented uniformly in NICUs across the United States. Concerns regarding an insufficient ROP workforce are validated.

Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability

PURPOSE

To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP).

DESIGN

We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD.

PARTICIPANTS

Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units.

METHODS

Expert classification of images of plus disease in ROP.

MAIN OUTCOME MEASURES

Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement).

RESULTS

There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0-0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R² = 0.82; and dataset B: P < 0.05 and adjusted R² = 0.6615).

CONCLUSIONS

There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis

PURPOSE

To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP.

DESIGN

We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases.

PARTICIPANTS

Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units.

METHODS

Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system.

MAIN OUTCOME MEASURES

Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling.

RESULTS

There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86).

CONCLUSIONS

Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

Diagnostic Discrepancies in Retinopathy of Prematurity Classification

PURPOSE

To identify the most common areas for discrepancy in retinopathy of prematurity (ROP) classification between experts.

DESIGN

Prospective cohort study.

PARTICIPANTS

A total of 281 infants were identified as part of a multicenter, prospective, ROP cohort study from 7 participating centers. Each site had participating ophthalmologists who provided the clinical classification after routine examination using binocular indirect ophthalmoscopy (BIO) and obtained wide-angle retinal images, which were independently classified by 2 study experts.

METHODS

Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were obtained from study subjects, and 2 experts evaluated each image using a secure web-based module. Image-based classifications for zone, stage, plus disease, and overall disease category (no ROP, mild ROP, type II or pre-plus, and type I) were compared between the 2 experts and with the clinical classification obtained by BIO.

MAIN OUTCOME MEASURES

Inter-expert image-based agreement and image-based versus ophthalmoscopic diagnostic agreement using absolute agreement and weighted kappa statistic.

RESULTS

A total of 1553 study eye examinations from 281 infants were included in the study. Experts disagreed on the stage classification in 620 of 1553 comparisons (40%), plus disease classification (including pre-plus) in 287 of 1553 comparisons (18%), zone in 117 of 1553 comparisons (8%), and overall ROP category in 618 of 1553 comparisons (40%). However, agreement for presence versus absence of type 1 disease was >95%. There were no differences between image-based and clinical classification except for zone III disease.

CONCLUSIONS

The most common area of discrepancy in ROP classification is stage, although inter-expert agreement for clinically significant disease, such as presence versus absence of type 1 and type 2 disease, is high. There were no differences between image-based grading and clinical examination in the ability to detect clinically significant disease. This study provides additional evidence that image-based classification of ROP reliably detects clinically significant levels of ROP with high accuracy compared with the clinical examination.

Genomics in the neonatal nursery: Focus on ROP

Retinopathy of prematurity (ROP) is a complex disease that is influenced by both genetic and environmental factors. Several small studies have found genetic variants in EPAS1, VEGF, SOD, and members of the WNT family in association with ROP. Design in genetic studies is challenging because of changing recommendations for the management of prematurity and ROP, the fact ROP is rare, and that availability of resources for managing premature infants can vary throughout the world. In addition, there is a shortage of ophthalmologists with the ability to diagnose and characterize severe ROP. Careful determination of the degree of prematurity is important when evaluating genetic studies. Controlling for significant epidemiologic factors and multiple comparisons is also important to consider when evaluating genetic studies. One large candidate gene study controlled for degree of prematurity, significant epidemiologic factors, and multiple comparisons and found variants within the intron of BDNF associated with severe ROP. Future studies using unbiased techniques to assess genetic risk are important as are in-depth study of BDNF through deep sequencing and associated mechanistic studies using appropriate experimental models.

Computer-Based Image Analysis for Plus Disease Diagnosis in Retinopathy of Prematurity: Performance of the "i-ROP" System and Image Features Associated With Expert Diagnosis

PURPOSE

We developed and evaluated the performance of a novel computer-based image analysis system for grading plus disease in retinopathy of prematurity (ROP), and identified the image features, shapes, and sizes that best correlate with expert diagnosis.

METHODS

A dataset of 77 wide-angle retinal images from infants screened for ROP was collected. A reference standard diagnosis was determined for each image by combining image grading from 3 experts with the clinical diagnosis from ophthalmoscopic examination. Manually segmented images were cropped into a range of shapes and sizes, and a computer algorithm was developed to extract tortuosity and dilation features from arteries and veins. Each feature was fed into our system to identify the set of characteristics that yielded the highest-performing system compared to the reference standard, which we refer to as the "i-ROP" system.

RESULTS

Among the tested crop shapes, sizes, and measured features, point-based measurements of arterial and venous tortuosity (combined), and a large circular cropped image (with radius 6 times the disc diameter), provided the highest diagnostic accuracy. The i-ROP system achieved 95% accuracy for classifying preplus and plus disease compared to the reference standard. This was comparable to the performance of the 3 individual experts (96%, 94%, 92%), and significantly higher than the mean performance of 31 nonexperts (81%).

CONCLUSIONS

This comprehensive analysis of computer-based plus disease suggests that it may be feasible to develop a fully-automated system based on wide-angle retinal images that performs comparably to expert graders at three-level plus disease discrimination.

TRANSLATIONAL RELEVANCE

Computer-based image analysis, using objective and quantitative retinal vascular features, has potential to complement clinical ROP diagnosis by ophthalmologists.

Inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity

PURPOSE

To evaluate inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity (ROP).

DESIGN

Prospective intra- and inter-rater reliability analysis.

METHODS

In this multicenter study, 260 wide-field digital photographs of 52 patients were presented to 7 recognized ROP experts on 2 consecutive assessment days 8 weeks apart. Experts were asked to assess the patients for ROP stage, presence of plus disease, presence of aggressive posterior ROP, necessity for treatment, and suggested treatment. Agreement levels were measured with Fleiss' kappa and Cohen's kappa.

RESULTS

Inter-expert agreement was fair for the ROP stage (κ = 0.24), plus disease (κ = 0.32), and aggressive posterior ROP (κ = 0.35); moderate for the necessity for treatment (κ = 0.41); and fair for the kind of treatment (κ = 0.38). Perfect inter-expert agreement was found in 9.6% of all patients for ROP stage 0-5, 45.1% for ≥ stage 2 ROP, 17.3% for plus disease, 57.7% for aggressive posterior ROP, and 25% for the necessity for treatment. Intra-expert agreement was higher than inter-expert agreement and was moderate for the ROP stage (κ = 0.56) and plus disease (κ = 0.51), moderate to substantial for aggressive posterior ROP (κ = 0.60), moderate for the necessity for treatment (κ = 0.47), and substantial for the kind of treatment (κ = 0.63).

CONCLUSIONS

ROP diagnosis and treatment decisions differ between experts and by 1 expert made on different days, indicating that the grading process is subjective and there is an observer bias when diagnosing ROP. These results could influence current practice in ROP assessment and training, and prompt further refinement of international ROP guidelines.

Global health training in ophthalmology residency programs

PURPOSE

To assess current global health education and international electives in ophthalmology residency programs and barriers to global health implementation in ophthalmology resident education.

METHODS

A web-based survey regarding participation in global health and international electives was emailed to residency program directors at 116 accredited ophthalmology residency programs via an Association of University Professors in Ophthalmology (AUPO) residency program director listserv.

RESULTS

Fifty-nine (51%) ophthalmology residency program directors responded. Thirty-seven program directors (63%) said global health was important to medical students when evaluating residency programs. Thirty-two program directors (55%) reported developing international electives. Reported barriers to resident participation in international electives were: 1) insufficient financial support, 2) inadequate resident coverage at home, and 3) lack of ACGME approval for international electives. Program directors requested more information about resident international electives, funding, and global ophthalmology educational resources. They requested ACGME recognition of international electives to facilitate resident participation. More than half (54%) of program directors supported international electives for residents.

CONCLUSIONS

This survey demonstrates that program directors believe global health is an important consideration when medical students evaluate training programs. Despite perceived barriers to incorporating global health opportunities into residency training, program directors are interested in development of global health resources and plan to further develop global health opportunities.

The Global Education Network for Retinopathy of Prematurity (Gen-Rop): Development, Implementation, and Evaluation of A Novel Tele-Education System (An American Ophthalmological Society Thesis)

PURPOSE

To describe the design, implementation, and evaluation of a tele-education system developed to improve diagnostic competency in retinopathy of prematurity (ROP) by ophthalmology residents.

METHODS

A secure Web-based tele-education system was developed utilizing a repository of over 2,500 unique image sets of ROP. For each image set used in the system, a reference standard ROP diagnosis was established. Performance by ophthalmology residents (postgraduate years 2 to 4) from the United States and Canada in taking the ROP tele-education program was prospectively evaluated. Residents were presented with image-based clinical cases of ROP during a pretest, posttest, and training chapters. Accuracy and reliability of ROP diagnosis (eg, plus disease, zone, stage, category) were determined using sensitivity, specificity, and the kappa statistic calculations of the results from the pretest and posttest.

RESULTS

Fifty-five ophthalmology residents were provided access to the ROP tele-education program. Thirty-one ophthalmology residents completed the program. When all training levels were analyzed together, a statistically significant increase was observed in sensitivity for the diagnosis of plus disease, zone, stage, category, and aggressive posterior ROP (P<.05). Statistically significant changes in specificity for identification of stage 2 or worse (P=.027) and pre-plus (P=.028) were observed.

CONCLUSIONS

A tele-education system for ROP education is effective in improving diagnostic accuracy of ROP by ophthalmology residents. This system may have utility in the setting of both healthcare and medical education reform by creating a validated method to certify telemedicine providers and educate the next generation of ophthalmologists.

Vascular endothelial growth factor antagonist therapy for retinopathy of prematurity

In this article, the growing problem of retinopathy of prematurity (ROP) worldwide, treatments for severe ROP including standard-of-care laser treatment, and the need for new treatments are discussed. Also discussed are the reasons to consider inhibiting the vascular endothelial growth factor (VEGF) signaling pathway in severe ROP and the concerns about broad VEGF inhibition. Finally, the potential role of VEGF in ROP based on studies in animal models of oxygen-induced retinopathy, the effects of anti-VEGF based on basic research data, and the clinical relevance of these data are covered.

Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology

Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.

Evaluation of an indirect ophthalmoscopy digital photographic system as a retinopathy of prematurity screening tool

PURPOSE

To determine whether digital retinal images obtained from an indirect ophthalmoscopy imaging system (Keeler) can be accurately graded for clinically significant retinopathy of prematurity (ROP) by masked experts.

METHODS

The medical records of infants screened for ROP who had posterior pole images acquired using the Keeler system during routine ROP examinations were retrospectively reviewed. Two reviewers, masked to patient demographics and clinical examination findings, graded the images for: (1) quality (good, fair, poor); (2) number of gradable quadrants, from 0 to 4; and (3) posterior pole disease (none, pre-plus, plus). The accuracy of grading Keeler images for clinically significant ROP (defined as pre-plus or plus disease) was compared to results of clinical examination.

RESULTS

One eye each of 253 infants was included. The mean postmenstrual age at examination was 35 weeks (range, 30-42). Grader 1 found the quality of 94% of images to be fair or good; grader 2, 83% of images. Grader 1 judged 87% of images to have ≥3 gradable quadrants; grader 2, 77% of images. The sensitivity and specificity of grading pre-plus or worse disease on Keeler images were 100% and 86%, respectively, for grader 1, and 94% and 89%, respectively, for grader 2.

CONCLUSIONS

Digital retinal images obtained by the Keeler system can be read with high sensitivity and specificity to screen for clinically important ROP. The Keeler system may be a valuable tool for ROP screening at remote locations (ie, via telemedicine).

The use of digital imaging in the identification of skip areas after laser treatment for retinopathy of prematurity and its implications for education and patient care

PURPOSE

To describe the utility of using wide-angle digital imaging in the training for retinopathy of prematurity with laser and in identifying common locations of skip areas that were present after initial panretinal photocoagulation with indirect ophthalmoscopy by ophthalmologists-in-training.

METHODS

Retrospective review of digital retinal images of 22 eyes of 12 infants who had undergone laser treatment for retinopathy of prematurity performed by ophthalmologists-in-training. Presence of skip areas was determined by masked review of photographs. The location of skip areas was classified based on two axes: 1) circumferential (in one of six clock-hour regions) and 2) radial (adjacent to the retinal ridge, adjacent to the ora serrata, or isolated patches of greater than one laser burn width).

RESULTS

A total of 30 skip areas were identified in the 22 eyes treated with laser photocoagulation. Based on the circumferential location, a significant difference in skip area distribution was found (P = 0.02). Regions with the highest percentage of skip areas were between the clock hours 11:00 to 1:00 (45%) and 5:00 to 7:00 (41%). Based on the radial location, 40% of all skip areas were found near the ora serrata, 17% near the ridge, and 43% as isolated patches (P = 0.14).

CONCLUSION

Skip areas after indirect panretinal laser photocoagulation by ophthalmologists-in-training were easily visualized by wide-angle digital imaging, after being missed by the trainee during the initial treatment procedure. Most skip areas in this study occurred in the superior or inferior retina. Digital imaging can assist ophthalmologists in visualizing all regions of the retina, can identify inadequate areas of laser treatment, and may reduce the need for retreatment after initial laser for retinopathy of prematurity.

Retinopathy of prematurity: a nationwide survey to evaluate current practices and preferences of ophthalmologists

PURPOSE

To emphasize the demographic and occupational characteristics of ophthalmologists primarily involved in retinopathy of prematurity (ROP) care and to ascertain their practices and preferences.

METHODS

A questionnaire was sent to all Turkish ophthalmologists known to be primarily involved in ROP. They were asked about personal and occupational characteristics, practices and preferences related to ROP care, and for their proposals related to ROP training.

RESULTS

Seventy questionnaires were sent, of which 63 (90%) were returned. A total of 44 respondents reported performing laser and 13 vitreoretinal surgery. Preferred treatment was transpupillary laser photocoagulation (54.5%), mostly performed in the operating theater (84.1%) under general anesthesia (72.7%). Only 19 (30.2%) reported intravitreal injection of vascular endothelial growth factor inhibitors (anti-VEGF). Ophthalmologists from university hospitals, trained in retina during fellowship, and performing treatment for ROP more commonly performed anti-VEGF injection. Most of them studied ROP after residency and 76.2% think that ROP training during residency is inadequate for clinical practice.

CONCLUSION

This survey reveals considerable variation among Turkish ROP specialists regarding ROP care. The survey data are critical in order to develop quality improvement and help in planning more effective future programs for ROP care in terms of training.

Pediatric ophthalmology fellowship training in laser ablation for retinopathy of prematurity

PURPOSE

To assess the training received by pediatric ophthalmology fellows in retinopathy of prematurity (ROP) laser ablation surgery.

METHODS

We surveyed recent graduates of 1-year pediatric ophthalmology fellowships to quantify the ROP laser training they received during their fellowship, the effect of fellowship training on their desire to treat ROP in practice, and interest in additional training modalities. Electronic questionnaires were distributed to all candidates-in-training of the American Association for Pediatric Ophthalmology and Strabismus.

RESULTS

The response rate was 64%, and 54 respondents met inclusion criteria. The mean number of ROP laser procedures performed during fellowship was 6.4; the median, 4. Eight respondents (15%) reported having performed no ROP laser procedures during their fellowship. Of the 54, 29 (54%) felt that they were at least adequately trained to perform ROP laser surgery and reported a mean of 10 (± 6.6) cases performed during fellowship. Respondents reporting that fellowship training was less than adequate performed a mean of 1.7 (± 1.5) cases (P < 0.01). A Pearson correlation coefficient of 0.8 revealed a moderately strong correlation between the amount of ROP laser training during fellowship and desire to perform ROP treatment in practice after fellowship. Twenty-six (48%) were very or somewhat interested in further ROP laser training.

CONCLUSIONS

The amount of laser experience during fellowship strongly influences the confidence in ROP laser skills of recently graduated pediatric ophthalmologists and the desire to perform laser procedures in their clinical practice. One-half of recently graduated pediatric ophthalmologists believe that their ROP laser training during fellowship was adequate for clinical practice.