Sensitivity and specificity of computer vision classification of eyelid photographs for programmatic trachoma assessment

Use of Deep Neural Networks in the Detection and Automated Classification of Lesions Using Clinical Images in Ophthalmology, Dermatology, and Oral Medicine-A Systematic Review

Artificial neural networks (ANN) are artificial intelligence (AI) techniques used in the automated recognition and classification of pathological changes from clinical images in areas such as ophthalmology, dermatology, and oral medicine. The combination of enterprise imaging and AI is gaining notoriety for its potential benefits in healthcare areas such as cardiology, dermatology, ophthalmology, pathology, physiatry, radiation oncology, radiology, and endoscopic. The present study aimed to analyze, through a systematic literature review, the application of performance of ANN and deep learning in the recognition and automated classification of lesions from clinical images, when comparing to the human performance. The PRISMA 2020 approach (Preferred Reporting Items for Systematic Reviews and Meta-analyses) was used by searching four databases of studies that reference the use of IA to define the diagnosis of lesions in ophthalmology, dermatology, and oral medicine areas. A quantitative and qualitative analyses of the articles that met the inclusion criteria were performed. The search yielded the inclusion of 60 studies. It was found that the interest in the topic has increased, especially in the last 3 years. We observed that the performance of IA models is promising, with high accuracy, sensitivity, and specificity, most of them had outcomes equivalent to human comparators. The reproducibility of the performance of models in real-life practice has been reported as a critical point. Study designs and results have been progressively improved. IA resources have the potential to contribute to several areas of health. In the coming years, it is likely to be incorporated into everyday life, contributing to the precision and reducing the time required by the diagnostic process.

Detection of trachoma using machine learning approaches

Background

Though significant progress in disease elimination has been made over the past decades, trachoma is the leading infectious cause of blindness globally. Further efforts in trachoma elimination are paradoxically being limited by the relative rarity of the disease, which makes clinical training for monitoring surveys difficult. In this work, we evaluate the plausibility of an Artificial Intelligence model to augment or replace human image graders in the evaluation/diagnosis of trachomatous inflammation—follicular (TF).

Methods

We utilized a dataset consisting of 2300 images with a 5% positivity rate for TF. We developed classifiers by implementing two state-of-the-art Convolutional Neural Network architectures, ResNet101 and VGG16, and applying a suite of data augmentation/oversampling techniques to the positive images. We then augmented our data set with additional images from independent research groups and evaluated performance.

Results

Models performed well in minimizing the number of false negatives, given the constraint of the low numbers of images in which TF was present. The best performing models achieved a sensitivity of 95% and positive predictive value of 50–70% while reducing the number images requiring skilled grading by 66–75%. Basic oversampling and data augmentation techniques were most successful at improving model performance, while techniques that are grounded in clinical experience, such as highlighting follicles, were less successful.

Discussion

The developed models perform well and significantly reduce the burden on graders by minimizing the number of false negative identifications. Further improvements in model skill will benefit from data sets with more TF as well as a range in image quality and image capture techniques used. While these models approach/meet the community-accepted standard for skilled field graders (i.e., Cohen’s Kappa >0.7), they are insufficient to be deployed independently/clinically at this time; rather, they can be utilized to significantly reduce the burden on skilled image graders.

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study (Preprint)

BACKGROUND

As trachoma is eliminated, skilled field graders become less adept at correctly identifying active disease (trachomatous inflammation-follicular [TF]). Deciding if trachoma has been eliminated from a district or if treatment strategies need to be continued or reinstated is of critical public health importance. Telemedicine solutions require both connectivity, which can be poor in the resource-limited regions of the world in which trachoma occurs, and accurate grading of the images.

OBJECTIVE

Our purpose was to develop and validate a cloud-based "virtual reading center" (VRC) model using crowdsourcing for image interpretation.

METHODS

The Amazon Mechanical Turk (AMT) platform was used to recruit lay graders to interpret 2299 gradable images from a prior field trial of a smartphone-based camera system. Each image received 7 grades for US $0.05 per grade in this VRC. The resultant data set was divided into training and test sets to internally validate the VRC. In the training set, crowdsourcing scores were summed, and the optimal raw score cutoff was chosen to optimize kappa agreement and the resulting prevalence of TF. The best method was then applied to the test set, and the sensitivity, specificity, kappa, and TF prevalence were calculated.

RESULTS

In this trial, over 16,000 grades were rendered in just over 60 minutes for US $1098 including AMT fees. After choosing an AMT raw score cut point to optimize kappa near the World Health Organization (WHO)-endorsed level of 0.7 (with a simulated 40% prevalence TF), crowdsourcing was 95% sensitive and 87% specific for TF in the training set with a kappa of 0.797. All 196 crowdsourced-positive images received a skilled overread to mimic a tiered reading center and specificity improved to 99%, while sensitivity remained above 78%. Kappa for the entire sample improved from 0.162 to 0.685 with overreads, and the skilled grader burden was reduced by over 80%. This tiered VRC model was then applied to the test set and produced a sensitivity of 99% and a specificity of 76% with a kappa of 0.775 in the entire set. The prevalence estimated by the VRC was 2.70% (95% CI 1.84%-3.80%) compared to the ground truth prevalence of 2.87% (95% CI 1.98%-4.01%).

CONCLUSIONS

A VRC model using crowdsourcing as a first pass with skilled grading of positive images was able to identify TF rapidly and accurately in a low prevalence setting. The findings from this study support further validation of a VRC and crowdsourcing for image grading and estimation of trachoma prevalence from field-acquired images, although further prospective field testing is required to determine if diagnostic characteristics are acceptable in real-world surveys with a low prevalence of the disease.

Trachoma

Trachoma is a neglected tropical disease caused by infection with conjunctival strains of Chlamydia trachomatis. It can result in blindness. Pathophysiologically, trachoma is a disease complex composed of two linked chronic processes: a recurrent, generally subclinical infectious-inflammatory disease that mostly affects children, and a non-communicable, cicatricial and, owing to trichiasis, eventually blinding disease that supervenes in some individuals later in life. At least 150 infection episodes over an individual's lifetime are needed to precipitate trichiasis; thus, opportunity exists for a just global health system to intervene to prevent trachomatous blindness. Trachoma is found at highest prevalence in the poorest communities of low-income countries, particularly in sub-Saharan Africa; in June 2021, 1.8 million people worldwide were going blind from the disease. Blindness attributable to trachoma can appear in communities many years after conjunctival C. trachomatis transmission has waned or ceased; therefore, the two linked disease processes require distinct clinical and public health responses. Surgery is offered to individuals with trichiasis and antibiotic mass drug administration and interventions to stimulate facial cleanliness and environmental improvement are designed to reduce infection prevalence and transmission. Together, these interventions comprise the SAFE strategy, which is achieving considerable success. Although much work remains, a continuing public health problem from trachoma in the year 2030 will be difficult for the world to excuse.

Evaluation of photography using head-mounted display technology (ICAPS) for district Trachoma surveys

Background

As the prevalence of trachoma declines worldwide, it is becoming increasingly expensive and challenging to standardize graders in the field for surveys to document elimination. Photography of the tarsal conjunctiva and remote interpretation may help alleviate these challenges. The purpose of this study was to develop, and field test an Image Capture and Processing System (ICAPS) to acquire hands-free images of the tarsal conjunctiva for upload to a virtual reading center for remote grading.

Methodology/Principal findings

This observational study was conducted during a district-level prevalence survey for trachomatous inflammation—follicular (TF) in Chamwino, Tanzania. The ICAPS was developed using a Samsung Galaxy S8 smartphone, a Samsung Gear VR headset, a foot pedal trigger and customized software allowing for hands-free photography. After a one-day training course, three trachoma graders used the ICAPS to collect images from 1305 children ages 1–9 years, which were expert-graded remotely for comparison with field grades. In our experience, the ICAPS was successful at scanning and assigning barcodes to images, focusing on the everted eyelid with adequate examiner hand visualization, and capturing images with sufficient detail to grade TF. The percentage of children with TF by photos and by field grade was 5%. Agreement between grading of the images compared to the field grades at the child level was kappa = 0.53 (95%CI = 0.40–0.66). There were ungradable images for at least one eye in 199 children (9.1%), with more occurring in children ages 1–3 (18.5%) than older children ages 4–9 (4.2%) (χ² = 145.3, p<0.001).

Conclusions/Significance

The prototype ICAPS device was robust, able to image 1305 children in a district level survey and transmit images from rural Tanzania to an online grading platform. More work is needed to improve the percentage of ungradable images and to better understand the causes of disagreement between field and photo grading.

Trachoma is the leading infectious cause of blindness worldwide, caused by the bacterium Chlamydia trachomatis. Programs targeting trachoma elimination in endemic regions largely rely on periodic prevalence surveys to monitor progress, but training field graders requires active cases, which is becoming challenging as prevalence declines. Photography of the tarsal conjunctiva with remote interpretation via telemedicine may serve as a more auditable, effective, and cost-efficient method for surveys. We developed and evaluated the Image Capture and Processing System (ICAPS), a smartphone-based, hands-free, head-mounted camera system (Samsung Galaxy S8 with custom app, Samsung Gear VR headset, and a Bluetooth-linked foot pedal trigger). The ICAPS was easy to use in challenging field conditions, was able to upload images from Tanzania and link images to field data. The percentage of TF was 5% by both field grade and photo grade, with agreement kappa = 0.53. Additional field training and enhanced certification of photographers may help reduce the proportion of ungradable images; further research on reasons for mismatch of grades between field and photo is needed.

Digital Transformation in Ophthalmic Clinical Care During the COVID-19 Pandemic

ABSTRACT

COVID-19 has placed unprecedented pressure on health systems globally, whereas simultaneously stimulating unprecedented levels of transformation. Here, we review digital adoption that has taken place during the pandemic to drive improvements in ophthalmic clinical care, with a specific focus on out-of-hospital triage and services, clinical assessment, patient management, and use of electronic health records. We show that although there have been some successes, shortcomings in technology infrastructure prepandemic became only more apparent and consequential as COVID-19 progressed. Through our review, we emphasize the need for clinicians to better grasp and harness key technology trends such as telecommunications and artificial intelligence, so that they can effectively and safely shape clinical practice using these tools going forward.

Trachoma Prevalence After Discontinuation of Mass Azithromycin Distribution

Background

As the World Health Organization seeks to eliminate trachoma by 2020, countries are beginning to control the transmission of trachomatous inflammation–follicular (TF) and discontinue mass drug administration (MDA) with oral azithromycin. We evaluated the effect of MDA discontinuation on TF_1–9 prevalence at the district level.

Methods

We extracted from the available data districts with an impact survey at the end of their program cycle that initiated discontinuation of MDA (TF_1–9 prevalence <5%), followed by a surveillance survey conducted to determine whether TF_1–9 prevalence remained below the 5% threshold, warranting discontinuation of MDA. Two independent analyses were performed, 1 regression based and 1 simulation based, that assessed the change in TF_1–9 from the impact survey to the surveillance survey.

Results

Of the 220 districts included, TF_1–9 prevalence increased to >5% from impact to surveillance survey in 9% of districts. Regression analysis indicated that impact survey TF_1–9 prevalence was a significant predictor of surveillance survey TF_1–9 prevalence. The proportion of simulations with >5% TF_1–9 prevalence in the surveillance survey was 2%, assuming the survey was conducted 4 years after MDA.

Conclusion

An increase in TF_1–9 prevalence may represent disease resurgence but could also be due to measurement error. Improved diagnostic tests are crucial to elimination of TF_1–9 as a public health problem.