PeriorbitAI: Artificial Intelligence Automation of Eyelid and Periorbital Measurements

Automatic labeling of facial zones for digital clinical application: An ensemble of semantic segmentation models

INTRODUCTION

The application of artificial intelligence to facial aesthetics has been limited by the inability to discern facial zones of interest, as defined by complex facial musculature and underlying structures. Although semantic segmentation models (SSMs) could potentially overcome this limitation, existing facial SSMs distinguish only three to nine facial zones of interest.

METHODS

We developed a new supervised SSM, trained on 669 high-resolution clinical-grade facial images; a subset of these images was used in an iterative process between facial aesthetics experts and manual annotators that defined and labeled 33 facial zones of interest.

RESULTS

Because some zones overlap, some pixels are included in multiple zones, violating the one-to-one relationship between a given pixel and a specific class (zone) required for SSMs. The full facial zone model was therefore used to create three sub-models, each with completely non-overlapping zones, generating three outputs for each input image that can be treated as standalone models. For each facial zone, the output demonstrating the best Intersection Over Union (IOU) value was selected as the winning prediction.

CONCLUSIONS

The new SSM demonstrates mean IOU values superior to manual annotation and landmark analyses, and it is more robust than landmark methods in handling variances in facial shape and structure.

Ensemble neural network model for detecting thyroid eye disease using external photographs

PURPOSE

To describe an artificial intelligence platform that detects thyroid eye disease (TED).

DESIGN

Development of a deep learning model.

METHODS

1944 photographs from a clinical database were used to train a deep learning model. 344 additional images ('test set') were used to calculate performance metrics. Receiver operating characteristic, precision-recall curves and heatmaps were generated. From the test set, 50 images were randomly selected ('survey set') and used to compare model performance with ophthalmologist performance. 222 images obtained from a separate clinical database were used to assess model recall and to quantitate model performance with respect to disease stage and grade.

RESULTS

The model achieved test set accuracy of 89.2%, specificity 86.9%, recall 93.4%, precision 79.7% and an F1 score of 86.0%. Heatmaps demonstrated that the model identified pixels corresponding to clinical features of TED. On the survey set, the ensemble model achieved accuracy, specificity, recall, precision and F1 score of 86%, 84%, 89%, 77% and 82%, respectively. 27 ophthalmologists achieved mean performance of 75%, 82%, 63%, 72% and 66%, respectively. On the second test set, the model achieved recall of 91.9%, with higher recall for moderate to severe (98.2%, n=55) and active disease (98.3%, n=60), as compared with mild (86.8%, n=68) or stable disease (85.7%, n=63).

CONCLUSIONS

The deep learning classifier is a novel approach to identify TED and is a first step in the development of tools to improve diagnostic accuracy and lower barriers to specialist evaluation.

Automated extraction of clinical measures from videos of oculofacial disorders using machine learning: feasibility, validity and reliability

OBJECTIVES

To determine the feasibility, validity and reliability of automatically extracting clinically meaningful eyelid measurements from consumer-grade videos of individuals with oculofacial disorders.

METHODS

A custom computer program was designed to automatically extract clinical measures from consumer-grade videos. This program was applied to publicly available videos of individuals with oculofacial disorders, and age-matched controls. The primary outcomes were margin reflex distance 1 (MRD1) and 2 (MRD2), blink lagophthalmos, and ocular surface area exposure. Test-retest reliability was evaluated using Bland-Altman analysis to compare the agreement in obtained measures between separate videos of the same individual taken within 48 h of each other.

RESULTS

MRD1 was reduced in individuals with ptosis versus controls (2.2 mm versus 3.4 mm, p < 0.001), and increased in individuals with facial nerve palsy (FNP) (3.9 mm, p = 0.049) and thyroid eye disease (TED) (4.1 mm; p = 0.038). Blink lagophthalmos was increased in individuals with FNP (3.7 mm); p < 0.001) and those with TED (0.1 mm, p = 0.003) versus controls (0.0 mm). Ocular surface exposure was reduced in individuals with ptosis compared with controls (12.2 mm2 versus 13.1 mm2; p < 0.001) and increased in TED (13.7 mm2; p 0.002). Bland-Altmann analysis demonstrated 95% limits of agreement for video-derived measures: median MRD1: -1.1 to 1.1 mm; median MRD2: -0.9 to 1.0 mm; blink lagophthalmos: -3.5 to 3.7 mm; and average ocular surface area exposure: -1.6 to 1.6 mm2.

CONCLUSIONS

The presented program is capable of taking consumer grade videos of patients with oculofacial disease and providing clinically meaningful and reliable eyelid measurements that show promising validity.

Toward safer ophthalmic artificial intelligence via distributed validation on real-world data

PURPOSE OF REVIEW

The current article provides an overview of the present approaches to algorithm validation, which are variable and largely self-determined, as well as solutions to address inadequacies.

RECENT FINDINGS

In the last decade alone, numerous machine learning applications have been proposed for ophthalmic diagnosis or disease monitoring. Remarkably, of these, less than 15 have received regulatory approval for implementation into clinical practice. Although there exists a vast pool of structured and relatively clean datasets from which to develop and test algorithms in the computational 'laboratory', real-world validation remains key to allow for safe, equitable, and clinically reliable implementation. Bottlenecks in the validation process stem from a striking paucity of regulatory guidance surrounding safety and performance thresholds, lack of oversight on critical postdeployment monitoring and context-specific recalibration, and inherent complexities of heterogeneous disease states and clinical environments. Implementation of secure, third-party, unbiased, pre and postdeployment validation offers the potential to address existing shortfalls in the validation process.

SUMMARY

Given the criticality of validation to the algorithm pipeline, there is an urgent need for developers, machine learning researchers, and end-user clinicians to devise a consensus approach, allowing for the rapid introduction of safe, equitable, and clinically valid machine learning implementations.

Artificial intelligence-assisted smartphone-based sensing for bioanalytical applications: A review

Artificial intelligence (AI) has received great attention since the concept was proposed, and it has developed rapidly in recent years with applications in many fields. Meanwhile, newer iterations of smartphone hardware technologies which have excellent data processing capabilities have leveraged on AI capabilities. Based on the desirability for portable detection, researchers have been investigating intelligent analysis by combining smartphones with AI algorithms. Various examples of the application of AI algorithm-based smartphone detection and analysis have been developed. In this review, we give an overview of this field, with a particular focus on bioanalytical detection applications. The applications are presented in terms of hardware design, software algorithms, and specific application areas. We also discuss the existing limitations of AI-based smartphone detection and analytical approaches, and their future prospects. The take-home message of our review is that the application of AI in the field of detection analysis is restricted by the limitations of the smartphone's hardware as well as the model building of AI for detection targets with insufficient data. Nevertheless, at this juncture, while bioanalytical diagnostics and health monitoring have set the pace for AI-based smartphone applicability, the future should see the technology making greater inroads into other fields. In relation to the latter, it is likely that the ordinary or average person will play a greater participatory role.

Training Deep Learning Models to Work on Multiple Devices by Cross-Domain Learning with No Additional Annotations

PURPOSE

To create an unsupervised cross-domain segmentation algorithm for segmenting intraretinal fluid and retinal layers on normal and pathologic macular OCT images from different manufacturers and camera devices.

DESIGN

We sought to use generative adversarial networks (GANs) to generalize a segmentation model trained on one OCT device to segment B-scans obtained from a different OCT device manufacturer in a fully unsupervised approach without labeled data from the latter manufacturer.

PARTICIPANTS

A total of 732 OCT B-scans from 4 different OCT devices (Heidelberg Spectralis, Topcon 1000, Maestro2, and Zeiss Plex Elite 9000).

METHODS

We developed an unsupervised GAN model, GANSeg, to segment 7 retinal layers and intraretinal fluid in Topcon 1000 OCT images (domain B) that had access only to labeled data on Heidelberg Spectralis images (domain A). GANSeg was unsupervised because it had access only to 110 Heidelberg labeled OCTs and 556 raw and unlabeled Topcon 1000 OCTs. To validate GANSeg segmentations, 3 masked graders manually segmented 60 OCTs from an external Topcon 1000 test dataset independently. To test the limits of GANSeg, graders also manually segmented 3 OCTs from Zeiss Plex Elite 9000 and Topcon Maestro2. A U-Net was trained on the same labeled Heidelberg images as baseline. The GANSeg repository with labeled annotations is at https://github.com/uw-biomedical-ml/ganseg.

MAIN OUTCOME MEASURES

Dice scores comparing segmentation results from GANSeg and the U-Net model with the manual segmented images.

RESULTS

Although GANSeg and U-Net achieved comparable Dice scores performance as human experts on the labeled Heidelberg test dataset, only GANSeg achieved comparable Dice scores with the best performance for the ganglion cell layer plus inner plexiform layer (90%; 95% confidence interval [CI], 68%-96%) and the worst performance for intraretinal fluid (58%; 95% CI, 18%-89%), which was statistically similar to human graders (79%; 95% CI, 43%-94%). GANSeg significantly outperformed the U-Net model. Moreover, GANSeg generalized to both Zeiss and Topcon Maestro2 swept-source OCT domains, which it had never encountered before.

CONCLUSIONS

GANSeg enables the transfer of supervised deep learning algorithms across OCT devices without labeled data, thereby greatly expanding the applicability of deep learning algorithms.

Development and Assessment of an Artificial Intelligence-Based Tool for Ptosis Measurement in Adult Myasthenia Gravis Patients Using Selfie Video Clips Recorded on Smartphones

Introduction

Myasthenia gravis (MG) is a rare autoimmune disease characterized by muscle weakness and fatigue. Ptosis (eyelid drooping) occurs due to fatigue of the muscles for eyelid elevation and is one symptom widely used by patients and healthcare providers to track progression of the disease. Margin reflex distance 1 (MRD1) is an accepted clinical measure of ptosis and is typically assessed using a hand-held ruler. In this work, we develop an AI model that enables automated measurement of MRD1 in self-recorded video clips collected using patient smartphones.

Methods

A 3-month prospective observational study collected a dataset of video clips from patients with MG. Study participants were asked to perform an eyelid fatigability exercise to elicit ptosis while filming "selfie" videos on their smartphones. These images were collected in nonclinical settings, with no in-person training. The dataset was annotated by non-clinicians for (1) eye landmarks to establish ground truth MRD1 and (2) the quality of the video frames. The ground truth MRD1 (in millimeters, mm) was calculated from eye landmark annotations in the video frames using a standard conversion factor, the horizontal visible iris diameter of the human eye. To develop the model, we trained a neural network for eye landmark detection consisting of a ResNet50 backbone plus two dense layers of 78 dimensions on publicly available datasets. Only the ResNet50 backbone was used, discarding the last two layers. The embeddings from the ResNet50 were used as features for a support vector regressor (SVR) using a linear kernel, for regression to MRD1, in mm. The SVR was trained on data collected remotely from MG patients in the prospective study, split into training and development folds. The model's performance for MRD1 estimation was evaluated on a separate test fold from the study dataset.

Results

On the full test fold (N = 664 images), the correlation between the ground truth and predicted MRD1 values was strong (r = 0.732). The mean absolute error was 0.822 mm; the mean of differences was -0.256 mm; and 95% limits of agreement (LOA) were -0.214-1.768 mm. Model performance showed no improvement when test data were gated to exclude "poor" quality images.

Conclusions

On data generated under highly challenging real-world conditions from a variety of different smartphone devices, the model predicts MRD1 with a strong correlation (r = 0.732) between ground truth and predicted MRD1.

Orbital and eyelid diseases: The next breakthrough in artificial intelligence?

Orbital and eyelid disorders affect normal visual functions and facial appearance, and precise oculoplastic and reconstructive surgeries are crucial. Artificial intelligence (AI) network models exhibit a remarkable ability to analyze large sets of medical images to locate lesions. Currently, AI-based technology can automatically diagnose and grade orbital and eyelid diseases, such as thyroid-associated ophthalmopathy (TAO), as well as measure eyelid morphological parameters based on external ocular photographs to assist surgical strategies. The various types of imaging data for orbital and eyelid diseases provide a large amount of training data for network models, which might be the next breakthrough in AI-related research. This paper retrospectively summarizes different imaging data aspects addressed in AI-related research on orbital and eyelid diseases, and discusses the advantages and limitations of this research field.