3D Approaches and Challenges in Facial Expression Recognition Algorithms—A Literature Review

Can smartphones be used for routine dental clinical application? A validation study for using smartphone-generated 3D facial images

OBJECTIVES

To compare the accuracy of smartphone-generated three-dimensional (3D) facial images to that of direct anthropometry (DA) and 3dMD with the aim of assessing the validity and reliability of smartphone-generated 3D facial images for routine clinical applications.

MATERIALS AND METHODS

Twenty-five anthropometric soft-tissue facial landmarks were labelled manually on 22 orthognathic surgery patients (11 males and 11 females; mean age 26.2 ± 5.3 years). For each labelled face, two imaging operations were performed using two different surface imaging systems: 3dMDface and Bellus3D FaceApp. Next, 42 inter-landmark facial measurements amongst the identified facial landmarks were measured directly on each labelled face and also digitally on 3D facial images. The measurements obtained from smartphone-generated 3D facial images (SGI) were statistically compared with those from DA and 3dMD.

RESULTS

SGI had slightly higher measurement values than DA and 3dMD, but there was no statistically significant difference between the mean values of inter-landmark measures across the three methods. Clinically acceptable differences (≤3 mm or ≤5°) were observed for 67 % and 74 % of measurements with good agreement between DA and SGI, and 3dMD and SGI, respectively. An overall small systematic bias of ± 0.2 mm was observed between the three methods. Furthermore, the mean absolute difference between DA and SGI methods was highest for linear (1.41 ± 0.33 mm) as well as angular measurements (3.07 ± 0.73°).

CONCLUSIONS

SGI demonstrated fair trueness compared to DA and 3dMD. The central region and flat areas of the face in SGI are more accurate. Despite this, SGI have limited clinical application, and the panfacial accuracy of the SGI would be more desirable from a clinical application standpoint.

CLINICAL SIGNIFICANCE

The usage of SGI in clinical practice for region-specific macro-proportional facial assessment involving central and flat regions of the face or for patient education purposes, which does not require accuracy within 3 mm and 5° can be considered.

Virtual Reality for Vision Science

Virtual reality (VR) allows us to create visual stimuli that are both immersive and reactive. VR provides many new opportunities in vision science. In particular, it allows us to present wide field-of-view, immersive visual stimuli; for observers to actively explore the environments that we create; and for us to understand how visual information is used in the control of behaviour. In contrast with traditional psychophysical experiments, VR provides much greater flexibility in creating environments and tasks that are more closely aligned with our everyday experience. These benefits of VR are of particular value in developing our theories of the behavioural goals of the visual system and explaining how visual information is processed to achieve these goals. The use of VR in vision science presents a number of technical challenges, relating to how the available software and hardware limit our ability to accurately specify the visual information that defines our virtual environments and the interpretation of data gathered in experiments with a freely moving observer in a responsive environment.

Movie Reviews Classification through Facial Image Recognition and Emotion Detection Using Machine Learning Methods

The critical component of HCI is face recognition technology. Emotional computing heavily relies on the identification of facial emotions. Applications for emotion-driven face animation and dynamic assessment are numerous (FER). Universities have started to support real-world face expression recognition research as a result. Short video clips are continually uploaded and shared online, building up a library of videos on various topics. The enormous amount of movie data appeals to system engineers and researchers of autonomous emotion mining and sentiment analysis. The main idea is that categorizing things may be done by looking at how individuals feel about specific issues. People might choose to have a basic or complex facial appearance. People worldwide continually express their feelings through their faces, whether they are happy, sad, or uncertain. An online user can visually express themselves through a video’s editing, music, and subtitles. Additionally, before the video data can be used, noise in the data must frequently be eliminated. Automatically figuring out how someone feels in a video is a challenging task that will only get harder over time. Therefore, this paper aims to show how facial recognition video analysis can be used to show how sentiment analysis can help with business growth and essential decision-making. To determine how people are affected by reviewers’ writing, we use a technique for deciding emotions in this analysis. The feelings in movies are assessed using machine learning algorithms to categorize them. A lightweight machine learning algorithm is proposed to help in Aspect-oriented emotion classification for movie reviews. Moreover, to analyze real and published datasets, experimental results are compared with different Machine Learning algorithms, i.e., Naive Bayes, Support Vector Machine, Random Forest, and CNN. The proposed approach obtained 84.72 accuracy and 79.24 sensitivity. Furthermore, the method has a specificity of 90.64 and a precision of 90.2. Thus, the proposed method significantly increases the accuracy and sensitivity of the emotion detection system from facial feature recognition. Our proposed algorithm has shown contribution to detect datasets of different emotions with symmetric characteristics and symmetrically-designed facial image recognition tasks.

Facial Emotion Recognition in Verbal Communication Based on Deep Learning

Facial emotion recognition from facial images is considered a challenging task due to the unpredictable nature of human facial expressions. The current literature on emotion classification has achieved high performance over deep learning (DL)-based models. However, the issue of performance degradation occurs in these models due to the poor selection of layers in the convolutional neural network (CNN) model. To address this issue, we propose an efficient DL technique using a CNN model to classify emotions from facial images. The proposed algorithm is an improved network architecture of its kind developed to process aggregated expressions produced by the Viola-Jones (VJ) face detector. The internal architecture of the proposed model was finalised after performing a set of experiments to determine the optimal model. The results of this work were generated through subjective and objective performance. An analysis of the results presented herein establishes the reliability of each type of emotion, along with its intensity and classification. The proposed model is benchmarked against state-of-the-art techniques and evaluated on the FER-2013, CK+, and KDEF datasets. The utility of these findings lies in their application by law-enforcing bodies in smart cities.

Design of Interactive Vocal Guidance and Artistic Psychological Intervention System Based on Emotion Recognition

The research on artistic psychological intervention to judge emotional fluctuations by extracting emotional features from interactive vocal signals has become a research topic with great potential for development. Based on the interactive vocal music instruction theory of emotion recognition, this paper studies the design of artistic psychological intervention system. This paper uses the vocal music emotion recognition algorithm to first train the interactive recognition network, in which the input is a row vector composed of different vocal music characteristics, and finally recognizes the vocal music of different emotional categories, which solves the problem of low data coupling in the artistic psychological intervention system. Among them, the vocal music emotion recognition experiment based on the interactive recognition network is mainly carried out from six aspects: the number of iterative training, the vocal music instruction rate, the number of emotion recognition signal nodes in the artistic psychological intervention layer, the number of sample sets, different feature combinations, and the number of emotion types. The input data of the system is a training class learning video, and actions and expressions need to be recognized before scoring. In the simulation process, before the completion of the sample indicators is unbalanced, the R language statistical analysis tool is used to balance the existing unbalanced data based on the artificial data synthesis method, and 279 uniformly classified samples are obtained. The 279∗7 dataset was used for statistical identification of the participants. The experimental results show that under the guidance of four different interactive vocal music, the vocal emotion recognition rate is between 65.85%-91.00%, which promotes the intervention of music therapy on artistic psychological intervention.

Robust facial expression recognition system in higher poses

Facial expression recognition (FER) has numerous applications in computer security, neuroscience, psychology, and engineering. Owing to its non-intrusiveness, it is considered a useful technology for combating crime. However, FER is plagued with several challenges, the most serious of which is its poor prediction accuracy in severe head poses. The aim of this study, therefore, is to improve the recognition accuracy in severe head poses by proposing a robust 3D head-tracking algorithm based on an ellipsoidal model, advanced ensemble of AdaBoost, and saturated vector machine (SVM). The FER features are tracked from one frame to the next using the ellipsoidal tracking model, and the visible expressive facial key points are extracted using Gabor filters. The ensemble algorithm (Ada-AdaSVM) is then used for feature selection and classification. The proposed technique is evaluated using the Bosphorus, BU-3DFE, MMI, CK + , and BP4D-Spontaneous facial expression databases. The overall performance is outstanding.

Continuous Emotion Recognition for Long-Term Behavior Modeling through Recurrent Neural Networks

One’s internal state is mainly communicated through nonverbal cues, such as facial expressions, gestures and tone of voice, which in turn shape the corresponding emotional state. Hence, emotions can be effectively used, in the long term, to form an opinion of an individual’s overall personality. The latter can be capitalized on in many human–robot interaction (HRI) scenarios, such as in the case of an assisted-living robotic platform, where a human’s mood may entail the adaptation of a robot’s actions. To that end, we introduce a novel approach that gradually maps and learns the personality of a human, by conceiving and tracking the individual’s emotional variations throughout their interaction. The proposed system extracts the facial landmarks of the subject, which are used to train a suitably designed deep recurrent neural network architecture. The above architecture is responsible for estimating the two continuous coefficients of emotion, i.e., arousal and valence, following the broadly known Russell’s model. Finally, a user-friendly dashboard is created, presenting both the momentary and the long-term fluctuations of a subject’s emotional state. Therefore, we propose a handy tool for HRI scenarios, where robot’s activity adaptation is needed for enhanced interaction performance and safety.

Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity

Emotion recognition is defined as identifying human emotion and is directly related to different fields such as human-computer interfaces, human emotional processing, irrational analysis, medical diagnostics, data-driven animation, human-robot communication, and many more. This paper proposes a new facial emotional recognition model using a convolutional neural network. Our proposed model, "ConvNet", detects seven specific emotions from image data including anger, disgust, fear, happiness, neutrality, sadness, and surprise. The features extracted by the Local Binary Pattern (LBP), region based Oriented FAST and rotated BRIEF (ORB) and Convolutional Neural network (CNN) from facial expressions images were fused to develop the classification model through training by our proposed CNN model (ConvNet). Our method can converge quickly and achieves good performance which the authors can develop a real-time schema that can easily fit the model and sense emotions. Furthermore, this study focuses on the mental or emotional stuff of a man or woman using the behavioral aspects. To complete the training of the CNN network model, we use the FER2013 databases at first, and then apply the generalization techniques to the JAFFE and CK+ datasets respectively in the testing stage to evaluate the performance of the model. In the generalization approach on the JAFFE dataset, we get a 92.05% accuracy, while on the CK+ dataset, we acquire a 98.13% accuracy which achieve the best performance among existing methods. We also test the system's success by identifying facial expressions in real-time. ConvNet consists of four layers of convolution together with two fully connected layers. The experimental results show that the ConvNet is able to achieve 96% training accuracy which is much better than current existing models. However, when compared to other validation methods, the suggested technique was more accurate. ConvNet also achieved validation accuracy of 91.01% for the FER2013 dataset. We also made all the materials publicly accessible for the research community at: https://github.com/Tanoy004/Emotion-recognition-through-CNN .

Towards Facial Gesture Recognition in Photographs of Patients with Facial Palsy

Humans express their emotions verbally and through actions, and hence emotions play a fundamental role in facial expressions and body gestures. Facial expression recognition is a popular topic in security, healthcare, entertainment, advertisement, education, and robotics. Detecting facial expressions via gesture recognition is a complex and challenging problem, especially in persons who suffer face impairments, such as patients with facial paralysis. Facial palsy or paralysis refers to the incapacity to move the facial muscles on one or both sides of the face. This work proposes a methodology based on neural networks and handcrafted features to recognize six gestures in patients with facial palsy. The proposed facial palsy gesture recognition system is designed and evaluated on a publicly available database with good results as a first attempt to perform this task in the medical field. We conclude that, to recognize facial gestures in patients with facial paralysis, the severity of the damage has to be considered because paralyzed organs exhibit different behavior than do healthy ones, and any recognition system must be capable of discerning these behaviors.

Research on a Real-Time Driver Fatigue Detection Algorithm Based on Facial Video Sequences

The research on driver fatigue detection is of great significance to improve driving safety. This paper proposes a real-time comprehensive driver fatigue detection algorithm based on facial landmarks to improve the detection accuracy, which detects the driver’s fatigue status by using facial video sequences without equipping their bodies with other intelligent devices. A tasks-constrained deep convolutional network is constructed to detect the face region based on 68 key points, which can solve the optimization problem caused by the different convergence speeds of each task. According to the real-time facial video images, the eye feature of the eye aspect ratio (EAR), mouth aspect ratio (MAR) and percentage of eye closure time (PERCLOS) are calculated based on facial landmarks. A comprehensive driver fatigue assessment model is established to assess the fatigue status of drivers through eye/mouth feature selection. After a series of comparative experiments, the results show that this proposed algorithm achieves good performance in both accuracy and speed for driver fatigue detection.

Face-Based Attention Recognition Model for Children with Autism Spectrum Disorder

Attention recognition plays a vital role in providing learning support for children with autism spectrum disorders (ASD). The unobtrusiveness of face-tracking techniques makes it possible to build automatic systems to detect and classify attentional behaviors. However, constructing such systems is a challenging task due to the complexity of attentional behavior in ASD. This paper proposes a face-based attention recognition model using two methods. The first is based on geometric feature transformation using a support vector machine (SVM) classifier, and the second is based on the transformation of time-domain spatial features to 2D spatial images using a convolutional neural network (CNN) approach. We conducted an experimental study on different attentional tasks for 46 children (ASD n=20, typically developing children n=26) and explored the limits of the face-based attention recognition model for participant and task differences. Our results show that the geometric feature transformation using an SVM classifier outperforms the CNN approach. Also, attention detection is more generalizable within typically developing children than within ASD groups and within low-attention tasks than within high-attention tasks. This paper highlights the basis for future face-based attentional recognition for real-time learning and clinical attention interventions.

Facial emotion mimicry in older adults with and without cognitive impairments due to Alzheimer's disease

Facial expression of humans is one of the main channels of everyday communication. The reported research work investigated communication regarding the pattern of emotional expression of healthy older adults and with mild cognitive impairments (MCI) or Alzheimer's disease (AD). It focuses on mimicking of displayed emotional facial expression on a sample of 25 older adults (healthy, MCI and AD patients). The adequacy of the patients' individual facial expressions in six basic emotions was measured with the Kinect 3D recording of the participants' facial expressions and compared to their own typical emotional facial expressions. The reactions were triggered by mimicking 49 still pictures of emotional facial expressions. No statistically significant differences in terms of frequency nor adequacy of emotional facial expression were reported in healthy and MCI groups. Unique patterns of emotional expressions have been observed in the AD group. Further investigating the pattern of older adults' facial expression may decrease the misunderstandings and increase the quality of life of the patients.

Lightweight and Resource-Constrained Learning Network for Face Recognition with Performance Optimization

Despite considerable progress in face recognition technology in recent years, deep learning (DL) and convolutional neural networks (CNN) have revealed commendable recognition effects with the advent of artificial intelligence and big data. FaceNet was presented in 2015 and is able to significantly improve the accuracy of face recognition, while also being powerfully built to counteract several common issues, such as occlusion, blur, illumination change, and different angles of head pose. However, not all hardware can sustain the heavy computing load in the execution of the FaceNet model. In applications in the security industry, lightweight and efficient face recognition are two key points for facilitating the deployment of DL and CNN models directly in field devices, due to their limited edge computing capability and low equipment cost. To this end, this paper provides a lightweight learning network improved from FaceNet, which is called FN13, to break through the hardware limitation of constrained computational resources. The proposed FN13 takes the advantage of center loss to reduce the variations of the between-class features and enlarge the difference of the within-class features, instead of the triplet loss by using FaceNet. The resulting model reduces the number of parameters and maintains a high degree of accuracy, only requiring few grayscale reference images per subject. The validity of FN13 is demonstrated by conducting experiments on the Labeled Faces in the Wild (LFW) dataset, as well as an analytical discussion regarding specific disguise problems.

Observing Pictures and Videos of Creative Products: An Eye Tracking Study

The paper offers insights into people’s exploration of creative products shown on a computer screen within the overall task of capturing artifacts’ original features and functions. In particular, the study presented here analyzes the effects of different forms of representations, i.e., static pictures and videos. While the relevance of changing stimuli’s forms of representation is acknowledged in both engineering design and human-computer interaction, scarce attention has been paid to this issue hitherto when creative products are in play. Six creative products have been presented to twenty-eight subjects through either pictures or videos in an Eye-Tracking-supported experiment. The results show that major attention is paid by people to original product features and functional elements when products are displayed by means of videos. This aspect is of paramount importance, as original shapes, parts, or characteristics of creative products might be inconsistent with people’s habits and cast doubts about their rationale and utility. In this sense, videos seemingly emphasize said original elements and likely lead to their explanation/resolution. Overall, the outcomes of the study strengthen the need to match appropriate forms of representation with different design stages in light of the needs for designs’ evaluation and testing user experience.

Questionnaires or Inner Feelings: Who Measures the Engagement Better?

This work proposes an innovative method for evaluating users’ engagement, combining the User Engagement Scale (UES) questionnaire and a facial expression recognition (FER) system, active research topics of increasing interest in the human–computer interaction domain (HCI). The subject of the study is a 3D simulator that reproduces a virtual FabLab in which users can approach and learn 3D modeling software and 3D printing. During the interaction with the virtual environment, a structured-light camera acquires the face of the participant in real-time, to catch its spontaneous reactions and compare them with the answers to the UES closed-ended questions. FER methods allow overcoming some intrinsic limits in the adoption of questioning methods, such as the non-sincerity of the interviewees and the lack of correspondence with facial expressions and body language. A convolutional neural network (CNN) has been trained on the Bosphorus database (DB) to perform expression recognition and the classification of the video frames in three classes of engagement (deactivation, average activation, and activation) according to the model of emotion developed by Russell. The results show that the two methodologies can be integrated to evaluate user engagement, to combine weighted answers and spontaneous reactions and to increase knowledge for the design of the new product or service.

Engagement Evaluation in a Virtual Learning Environment via Facial Expression Recognition and Self-Reports: A Preliminary Approach

Due to its versatility, virtual technology is being widely employed in different domains, from industry to amusement. The possibility to adopt this technology in early product/service design is going to bring positive effects such as the reduction of costs associated with the production of physical prototypes and the generation of a more effective knowledge of users’ feedback. This study proposes a preliminary methodology to evaluate users’ engagement in interacting with a virtual environment that consists of the integration between a self-report method (the user engagement scale questionnaire) and a method based on facial expression recognition. Results reported in this paper show that the two methodologies generate different types of knowledge which can be used to fit users’ needs and expectations. Even if focused on a specific case study, i.e., the evaluation of the engagement in a virtual learning environment, this paper aims to propose a methodology that can be applied to all kinds of virtual products.

Perspective Morphometric Criteria for Facial Beauty and Proportion Assessment

Common sense usually considers the assessment of female human attractiveness to be subjective. Nevertheless, in the past decades, several studies and experiments showed that an objective component in beauty assessment exists and can be strictly related, even if it does not match, with proportions of features. Proportions can be studied through analysis of the face, which relies on landmarks, i.e., specific points on the facial surface, which are shared by everyone, and measurements between them. In this work, several measures have been gathered from studies in the literature considering datasets of beautiful women to build a set of measures that can be defined as suggestive of female attractiveness. The resulting set consists of 29 measures applied to a public dataset, the Bosphorus database, whose faces have been both analyzed by the developed methodology based on the expanded set of measures and judged by human observers. Results show that the set of chosen measures is significant in terms of attractiveness evaluation, confirming the key role of proportions in beauty assessment; furthermore, the sorting of identified measures has been performed to identify the most significant canons involved in the evaluation.

StoolNet for Color Classification of Stool Medical Images

The color classification of stool medical images is commonly used to diagnose digestive system diseases, so it is important in clinical examination. In order to reduce laboratorians’ heavy burden, advanced digital image processing technologies and deep learning methods are employed for the automatic color classification of stool images in this paper. The region of interest (ROI) is segmented automatically and then classified with a shallow convolutional neural network (CNN) dubbed StoolNet. Thanks to its shallow structure and accurate segmentation, StoolNet can converge quickly. The sufficient experiments confirm the good performance of StoolNet and the impact of the different training sample numbers on StoolNet. The proposed method has several advantages, such as low cost, accurate automatic segmentation, and color classification. Therefore, it can be widely used in artificial intelligence (AI) healthcare.