Evaluation of photography using head-mounted display technology (ICAPS) for district Trachoma surveys

Computer Vision Identification of Trachomatous Inflammation-Follicular Using Deep Learning

PURPOSE

Trachoma surveys are used to estimate the prevalence of trachomatous inflammation-follicular (TF) to guide mass antibiotic distribution. These surveys currently rely on human graders, introducing a significant resource burden and potential for human error. This study describes the development and evaluation of machine learning models intended to reduce cost and improve reliability of these surveys.

METHODS

Fifty-six thousand seven hundred twenty-five everted eyelid photographs were obtained from 11,358 children of age 0 to 9 years in a single trachoma-endemic region of Ethiopia over a 3-year period. Expert graders reviewed all images from each examination to determine the estimated number of tarsal conjunctival follicles and the degree of trachomatous inflammation-intense. The median estimate of the 3 grader groups was used as the ground truth to train a MobileNetV3 large deep convolutional neural network to detect cases with TF.

RESULTS

The classification model predicted a TF prevalence of 32%, which was not significantly different from the human consensus estimate (30%; 95% confidence interval of difference, -2 to +4%). The model had an area under the receiver operating characteristic curve of 0.943, F1 score of 0.923, 88% accuracy, 83% sensitivity, and 91% specificity. The area under the receiver operating characteristic curve increased to 0.995 when interpreting nonborderline cases of TF.

CONCLUSIONS

Deep convolutional neural network models performed well at classifying TF and detecting the number of follicles evident in conjunctival photographs. Implementation of similar models may enable accurate, efficient, large-scale trachoma screening. Further validation in diverse populations with varying TF prevalence is needed before implementation at scale.

Accuracy, acceptability and feasibility of photography for use in trachoma surveys: a mixed methods study in Tanzania

BACKGROUND

Photography could be used to train individuals to diagnose trachomatous inflammation-follicular (TF) as trachoma prevalence decreases and to ensure accurate field TF grading in trachoma prevalence surveys. We compared photograph and field TF grading and determined the acceptability and feasibility of eyelid photography to community members and trachoma survey trainers.

METHODS

A total of 100 children ages 1-9 y were examined for TF in two Maasai villages in Tanzania. Two images of the right everted superior tarsal conjunctiva of each child were taken with a smartphone and a digital single-lens reflex (DSLR) camera. Two graders independently graded all photos. Focus group discussions (FGDs) were conducted with community members and Tropical Data trainers.

RESULTS

Of 391 photos, one-fifth were discarded as ungradable. Compared with field grading, photo grading consistently underdiagnosed TF. Compared with field grading, DSLR photo grading resulted in a higher prevalence and sensitivity than smartphone photo grading. FGDs indicated that communities and trainers found photography acceptable and preferred smartphones to DSLR in terms of practicalities, but image quality was of paramount importance for trainers.

CONCLUSIONS

Photography is acceptable and feasible, but further work is needed to ensure high-quality images that enable accurate and consistent grading before being routinely implemented in trachoma surveys.

Comparing image quality and trachoma detection across three camera types from a survey in Kongwa, Tanzania

BACKGROUND

There is an increasing demand for photography for trachoma prevalence surveys. In previous studies, digital single lens reflex (DSLR) images were superior to smartphone images, but newer-model smartphones and/or lens attachments may be able to bridge this gap. This study compares the image quality and ability to detect trachomatous inflammation - follicular (TF) of three camera types: a DSLR Nikon camera, an iPhone SE and an iPhone 13 Pro with a cell scope.

METHODS

We surveyed 62 children ages 1-7 y from two Tanzanian communities. Upper tarsal conjunctiva images of both eyes were graded for TF by two standardized graders. The McNemar's test and a logistic regression model were used for analyses.

RESULTS

The DSLR camera malfunctioned during the study, thus the iPhone SE and iPhone 13 Pro with cell scope were both more likely to take high-quality, gradable photographs (88% and 86%, respectively) compared with the DSLR camera (69%) (p<0.001 and p=0.02, respectively). TF was detected in gradable images from the iPhone SE (8.8%) and iPhone 13 Pro with cell scope (9.0%) at the same rate (p=1.0) as images from the DSLR camera (9.7%).

CONCLUSION

Smartphones with high-quality image capture, like the iPhone SE/13 Pro, have the potential for use in trachoma surveys if the proportion of gradable images can be improved.

Tropical Data: Approach and Methodology as Applied to Trachoma Prevalence Surveys

PURPOSE

Population-based prevalence surveys are essential for decision-making on interventions to achieve trachoma elimination as a public health problem. This paper outlines the methodologies of Tropical Data, which supports work to undertake those surveys.

METHODS

Tropical Data is a consortium of partners that supports health ministries worldwide to conduct globally standardised prevalence surveys that conform to World Health Organization recommendations. Founding principles are health ministry ownership, partnership and collaboration, and quality assurance and quality control at every step of the survey process. Support covers survey planning, survey design, training, electronic data collection and fieldwork, and data management, analysis and dissemination. Methods are adapted to meet local context and needs. Customisations, operational research and integration of other diseases into routine trachoma surveys have also been supported.

RESULTS

Between 29th February 2016 and 24th April 2023, 3373 trachoma surveys across 50 countries have been supported, resulting in 10,818,502 people being examined for trachoma.

CONCLUSION

This health ministry-led, standardised approach, with support from the start to the end of the survey process, has helped all trachoma elimination stakeholders to know where interventions are needed, where interventions can be stopped, and when elimination as a public health problem has been achieved. Flexibility to meet specific country contexts, adaptation to changes in global guidance and adjustments in response to user feedback have facilitated innovation in evidence-based methodologies, and supported health ministries to strive for global disease control targets.

Comparison of Five Camera Systems for Capturing and Grading Trachoma Images

PURPOSE

As training of trachoma graders using live participants grows increasingly difficult and expensive, alternative ways are needed possibly through replacement of field grading with photography. However, minimum specifications for a camera system capable of capturing high quality images have not been defined. This study compared images captured using four smartphones with those from a Nikon SLR camera for image quality and assessment of trachomatous inflammation - follicular (TF).

METHODS

The smartphones - Samsung Galaxy S8 (S8), Techno Camon 17 pro (TC), Infinix Note 10 pro (IN), Huawei p30 pro (HP) - were chosen for their availability and likelihood of good performance based on specifications without external attachments. All smartphones were used in random order for each participant.

RESULTS

129 children in Kongwa, Tanzania were enrolled (32.8% TF prevalence). The SLR camera had the least percent of ungradable images (3.1%), followed by the S8 (14%), HP (23.4%), IN (65.9%), and TC (71.2%). The S8 and the HP were significantly more likely to take ungradable images if they were used toward the end of the camera rotation. Agreement between the SLR and field grade was kappa = 0.73. Agreement between the field grade and gradable images from the S8 (0.68) and HP (0.8) was measured.

CONCLUSIONS

Published specifications did not predict the success of using different smartphones for everted eyelid photographs; proprietary post-processing software likely influenced gradeability. Smartphones, though we cannot recommend those tested in this study, may be viable for capturing images for trachoma provided the quality of images from the field are adequate.

Photographic grading for trachoma diagnosis within trachoma impact surveys in Amhara region, Ethiopia

BACKGROUND

As countries reach the trachoma elimination threshold and cases of trachomatous inflammation follicular (TF) become rare, it becomes difficult to train survey graders to recognize clinical signs. We assess the use of photography as a grading tool, the efficiency of an in-country grading center and the comparability of field and photographic grading.

METHODS

During January-February 2017 surveys in Amhara, Ethiopia, field graders assessed TF, trachomatous inflammation intense (TI) and trachomatous scarring (TS). Photographs were taken from each conjunctiva and later graded at the Gondar Grading Center (GGC) at the University of Gondar in Amhara. Two trained ophthalmology residents graded each set of photographs and a third grader provided an adjudicating grade when needed.

RESULTS

A total of 4953 photographs of 2477 conjunctivae from 1241 participants in 10 communities were graded over 5 d at the GGC. Six examined participants were not photographed. Agreement between field and photographic grades were for TF: percent agreement (PA) 96.7%, κ=0.70 (95% confidence interval [CI] 0.64 to 0.77; for TI: PA 94.7%, κ=0.32 (95% CI 0.20 to 0.43); and for TS: PA 83.5%, κ=0.22 (95% CI 0.15 to 0.29).

CONCLUSIONS

Conjunctival photography may be a solution for programs near the elimination threshold where there are few available community cases for training field graders.

Detection of trachoma using machine learning approaches

Background

Though significant progress in disease elimination has been made over the past decades, trachoma is the leading infectious cause of blindness globally. Further efforts in trachoma elimination are paradoxically being limited by the relative rarity of the disease, which makes clinical training for monitoring surveys difficult. In this work, we evaluate the plausibility of an Artificial Intelligence model to augment or replace human image graders in the evaluation/diagnosis of trachomatous inflammation—follicular (TF).

Methods

We utilized a dataset consisting of 2300 images with a 5% positivity rate for TF. We developed classifiers by implementing two state-of-the-art Convolutional Neural Network architectures, ResNet101 and VGG16, and applying a suite of data augmentation/oversampling techniques to the positive images. We then augmented our data set with additional images from independent research groups and evaluated performance.

Results

Models performed well in minimizing the number of false negatives, given the constraint of the low numbers of images in which TF was present. The best performing models achieved a sensitivity of 95% and positive predictive value of 50–70% while reducing the number images requiring skilled grading by 66–75%. Basic oversampling and data augmentation techniques were most successful at improving model performance, while techniques that are grounded in clinical experience, such as highlighting follicles, were less successful.

Discussion

The developed models perform well and significantly reduce the burden on graders by minimizing the number of false negative identifications. Further improvements in model skill will benefit from data sets with more TF as well as a range in image quality and image capture techniques used. While these models approach/meet the community-accepted standard for skilled field graders (i.e., Cohen’s Kappa >0.7), they are insufficient to be deployed independently/clinically at this time; rather, they can be utilized to significantly reduce the burden on skilled image graders.

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study (Preprint)

BACKGROUND

As trachoma is eliminated, skilled field graders become less adept at correctly identifying active disease (trachomatous inflammation-follicular [TF]). Deciding if trachoma has been eliminated from a district or if treatment strategies need to be continued or reinstated is of critical public health importance. Telemedicine solutions require both connectivity, which can be poor in the resource-limited regions of the world in which trachoma occurs, and accurate grading of the images.

OBJECTIVE

Our purpose was to develop and validate a cloud-based "virtual reading center" (VRC) model using crowdsourcing for image interpretation.

METHODS

The Amazon Mechanical Turk (AMT) platform was used to recruit lay graders to interpret 2299 gradable images from a prior field trial of a smartphone-based camera system. Each image received 7 grades for US $0.05 per grade in this VRC. The resultant data set was divided into training and test sets to internally validate the VRC. In the training set, crowdsourcing scores were summed, and the optimal raw score cutoff was chosen to optimize kappa agreement and the resulting prevalence of TF. The best method was then applied to the test set, and the sensitivity, specificity, kappa, and TF prevalence were calculated.

RESULTS

In this trial, over 16,000 grades were rendered in just over 60 minutes for US $1098 including AMT fees. After choosing an AMT raw score cut point to optimize kappa near the World Health Organization (WHO)-endorsed level of 0.7 (with a simulated 40% prevalence TF), crowdsourcing was 95% sensitive and 87% specific for TF in the training set with a kappa of 0.797. All 196 crowdsourced-positive images received a skilled overread to mimic a tiered reading center and specificity improved to 99%, while sensitivity remained above 78%. Kappa for the entire sample improved from 0.162 to 0.685 with overreads, and the skilled grader burden was reduced by over 80%. This tiered VRC model was then applied to the test set and produced a sensitivity of 99% and a specificity of 76% with a kappa of 0.775 in the entire set. The prevalence estimated by the VRC was 2.70% (95% CI 1.84%-3.80%) compared to the ground truth prevalence of 2.87% (95% CI 1.98%-4.01%).

CONCLUSIONS

A VRC model using crowdsourcing as a first pass with skilled grading of positive images was able to identify TF rapidly and accurately in a low prevalence setting. The findings from this study support further validation of a VRC and crowdsourcing for image grading and estimation of trachoma prevalence from field-acquired images, although further prospective field testing is required to determine if diagnostic characteristics are acceptable in real-world surveys with a low prevalence of the disease.