Meticulously detailed eye region model and its application to analysis of facial images

Application of artificial intelligence in oculoplastics

Oculoplastics is a subspecialty of ophthalmology/dermatology concerned with eyelid, orbital, and lacrimal diseases. Artificial intelligence (AI), with its powerful ability to analyze large data sets, has dramatically benefited oculoplastics. The cutting-edge AI technology is widely applied to extract ocular parameters and to use these results for further assessment, such as screening and diagnosis of blepharoptosis and predicting the progression of thyroid eye disease. AI also assists in treatment procedures, such as surgical strategy planning in blepharoptosis. High efficiency and high reliability are the most apparent advantages of AI, with promising prospects. The possibilities of AI in oculoplastics may lie in three-dimensional modeling technology and image generation. We retrospectively summarize AI applications involving eyelid, orbital, and lacrimal diseases in oculoplastics, and we also examine the strengths and weaknesses of AI technology in oculoplastics.

Orbital and eyelid diseases: The next breakthrough in artificial intelligence?

Orbital and eyelid disorders affect normal visual functions and facial appearance, and precise oculoplastic and reconstructive surgeries are crucial. Artificial intelligence (AI) network models exhibit a remarkable ability to analyze large sets of medical images to locate lesions. Currently, AI-based technology can automatically diagnose and grade orbital and eyelid diseases, such as thyroid-associated ophthalmopathy (TAO), as well as measure eyelid morphological parameters based on external ocular photographs to assist surgical strategies. The various types of imaging data for orbital and eyelid diseases provide a large amount of training data for network models, which might be the next breakthrough in AI-related research. This paper retrospectively summarizes different imaging data aspects addressed in AI-related research on orbital and eyelid diseases, and discusses the advantages and limitations of this research field.

CNN-Based Pupil Center Detection for Wearable Gaze Estimation System

This paper presents a convolutional neural network- (CNN-) based pupil center detection method for a wearable gaze estimation system using infrared eye images. Potentially, the pupil center position of a user’s eye can be used in various applications, such as human-computer interaction, medical diagnosis, and psychological studies. However, users tend to blink frequently; thus, estimating gaze direction is difficult. The proposed method uses two CNN models. The first CNN model is used to classify the eye state and the second is used to estimate the pupil center position. The classification model filters images with closed eyes and terminates the gaze estimation process when the input image shows a closed eye. In addition, this paper presents a process to create an eye image dataset using a wearable camera. This dataset, which was used to evaluate the proposed method, has approximately 20,000 images and a wide variation of eye states. We evaluated the proposed method from various perspectives. The result shows that the proposed method obtained good accuracy and has the potential for application in wearable device-based gaze estimation.

Robust Eye Center Localization through Face Alignment and Invariant Isocentric Patterns

The localization of eye centers is a very useful cue for numerous applications like face recognition, facial expression recognition, and the early screening of neurological pathologies. Several methods relying on available light for accurate eye-center localization have been exploited. However, despite the considerable improvements that eye-center localization systems have undergone in recent years, only few of these developments deal with the challenges posed by the profile (non-frontal face). In this paper, we first use the explicit shape regression method to obtain the rough location of the eye centers. Because this method extracts global information from the human face, it is robust against any changes in the eye region. We exploit this robustness and utilize it as a constraint. To locate the eye centers accurately, we employ isophote curvature features, the accuracy of which has been demonstrated in a previous study. By applying these features, we obtain a series of eye-center locations which are candidates for the actual position of the eye-center. Among these locations, the estimated locations which minimize the reconstruction error between the two methods mentioned above are taken as the closest approximation for the eye centers locations. Therefore, we combine explicit shape regression and isophote curvature feature analysis to achieve robustness and accuracy, respectively. In practical experiments, we use BioID and FERET datasets to test our approach to obtaining an accurate eye-center location while retaining robustness against changes in scale and pose. In addition, we apply our method to non-frontal faces to test its robustness and accuracy, which are essential in gaze estimation but have seldom been mentioned in previous works. Through extensive experimentation, we show that the proposed method can achieve a significant improvement in accuracy and robustness over state-of-the-art techniques, with our method ranking second in terms of accuracy. According to our implementation on a PC with a Xeon 2.5Ghz CPU, the frame rate of the eye tracking process can achieve 38 Hz.

Salient and Non-Salient Fiducial Detection using a Probabilistic Graphical Model

Deformable shape detection is an important problem in computer vision and pattern recognition. However, standard detectors are typically limited to locating only a few salient landmarks such as landmarks near edges or areas of high contrast, often conveying insufficient shape information. This paper presents a novel statistical pattern recognition approach to locate a dense set of salient and non-salient landmarks in images of a deformable object. We explore the fact that several object classes exhibit a homogeneous structure such that each landmark position provides some information about the position of the other landmarks. In our model, the relationship between all pairs of landmarks is naturally encoded as a probabilistic graph. Dense landmark detections are then obtained with a new sampling algorithm that, given a set of candidate detections, selects the most likely positions as to maximize the probability of the graph. Our experimental results demonstrate accurate, dense landmark detections within and across different databases.

Robust video-oculography for non-invasive autonomic nerve quantification

A relationship between autonomic nerves activity and depression or Alzheimer's disease has been reported. The quantification of autonomic nerves is expected to serve as a tool for quantifying the of severity of the disease or for early detection. Video-oculography is known as a non-invasive and reliable procedure of measurement of pupil response and is used in clinical practice. However, measuring the transition of pupil areas accurately is often difficult due to eyelid overlap, effects of blinking, eyelashes etc. Current video-oculography only performs thresholding to split pupil area and backgrounds and therefore sometimes has difficult in measuring accurate transitions of pupil reflex. In this study, we developed a robust and accurate method to measure the transition of pupil size. The proposed method introduces an interpolation process using an active contour model and ellipse estimation with selection of reliable contour points and attains robust measurement of pupil area against the abovementioned difficulties. We confirmed our method achieved an extraction accuracy of 98.3 % in precision and 98.9% in recall in average on the tested a total of 8,518 image frames from 30 movies.

Features versus context: An approach for precise and detailed detection and delineation of faces and facial features

The appearance-based approach to face detection has seen great advances in the last several years. In this approach, we learn the image statistics describing the texture pattern (appearance) of the object class we want to detect, e.g., the face. However, this approach has had limited success in providing an accurate and detailed description of the internal facial features, i.e., eyes, brows, nose, and mouth. In general, this is due to the limited information carried by the learned statistical model. While the face template is relatively rich in texture, facial features (e.g., eyes, nose, and mouth) do not carry enough discriminative information to tell them apart from all possible background images. We resolve this problem by adding the context information of each facial feature in the design of the statistical model. In the proposed approach, the context information defines the image statistics most correlated with the surroundings of each facial component. This means that when we search for a face or facial feature, we look for those locations which most resemble the feature yet are most dissimilar to its context. This dissimilarity with the context features forces the detector to gravitate toward an accurate estimate of the position of the facial feature. Learning to discriminate between feature and context templates is difficult, however, because the context and the texture of the facial features vary widely under changing expression, pose, and illumination, and may even resemble one another. We address this problem with the use of subclass divisions. We derive two algorithms to automatically divide the training samples of each facial feature into a set of subclasses, each representing a distinct construction of the same facial component (e.g., closed versus open eyes) or its context (e.g., different hairstyles). The first algorithm is based on a discriminant analysis formulation. The second algorithm is an extension of the AdaBoost approach. We provide extensive experimental results using still images and video sequences for a total of 3,930 images. We show that the results are almost as good as those obtained with manual detection.

All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous

We investigated the correspondence between perceived meanings of smiles and their morphological and dynamic characteristics. Morphological characteristics included co-activation of Orbicularis oculi (AU 6), smile controls, mouth opening, amplitude, and asymmetry of amplitude. Dynamic characteristics included duration, onset and offset velocity, asymmetry of velocity, and head movements. Smile characteristics were measured using the Facial Action Coding System (Ekman, Friesen, & Hager, 2002) and Automated Facial Image Analysis (Cohn & Kanade, 2007). Observers judged 122 smiles as amused, embarrassed, nervous, polite, or other. Fifty-three smiles met criteria for classification as perceived amused, embarrassed/nervous, or polite. In comparison with perceived polite, perceived amused more often included AU 6, open mouth, smile controls, larger amplitude, larger maximum onset and offset velocity, and longer duration. In comparison with perceived embarrassed/nervous, perceived amused more often included AU 6, lower maximum offset velocity, and smaller forward head pitch. In comparison with perceived polite, perceived embarrassed more often included mouth opening and smile controls, larger amplitude, and greater forward head pitch. Occurrence of the AU 6 in perceived embarrassed/nervous and polite smiles questions the assumption that AU 6 with a smile is sufficient to communicate felt enjoyment. By comparing three perceptually distinct types of smiles, we found that perceived smile meanings were related to specific variation in smile morphological and dynamic characteristics.

Infant Smiling Dynamics and Perceived Positive Emotion

To better understand early positive emotional expression, automated software measurements of facial action were supplemented with anatomically based manual coding. These convergent measurements were used to describe the dynamics of infant smiling and predict perceived positive emotional intensity. Over the course of infant smiles, degree of smile strength varied with degree of eye constriction (cheek raising, the Duchenne marker), which varied with degree of mouth opening. In a series of three rating studies, automated measurements of smile strength and mouth opening predicted naïve (undergraduate) observers' continuous ratings of video clips of smile sequences, as well as naïve and experienced (parent) ratings of positive emotion in still images from the sequences. An a priori measure of smile intensity combining anatomically based manual coding of both smile strength and mouth opening predicted positive emotion ratings of the still images. The findings indicate the potential of automated and fine-grained manual measurements of facial actions to describe the course of emotional expressions over time and to predict perceptions of emotional intensity.

ADAPTIVE APPEARANCE BASED FACE RECOGNITION

In this paper, we present an adaptive appearance based face recognition framework that combines the efficiency of global approaches and the robustness of local approaches together. The framework uses a novel eye locator to select an appropriate scheme for appearance based recognition. The eye locator first locates eye candidates via a new strength assignment, determined by the dissimilarity between the local appearance of an image point and the appearance of its neighboring points. Then the eye locator applies a simple but flexible model (half-circle snake) to the local context of the eye candidates in order to either refine the location of an eye candidate or discard non-eye candidates. We show the performance of our framework by testing on challenging face datasets containing extreme expressions, severe occlusions, and varied lighting conditions.