Emotional context influences micro-expression recognition

Horses (Equus caballus) facial micro-expressions: insight into discreet social information

Facial micro-expressions are facial expressions expressed briefly (less than 500 ms) and involuntarily. Described only in humans, we investigated whether micro-expressions could also be expressed by non-human animal species. Using the Equine Facial action coding system (EquiFACS), an objective tool based on facial muscles actions, we demonstrated that a non-human species, Equus caballus, is expressing facial micro-expressions in a social context. The AU17, AD38 and AD1 were selectively modulated as micro-expression-but not as standard facial expression (all durations included)-in presence of a human experimenter. As standard facial expressions, they have been associated with pain or stress but our results didn't support this association for micro-expressions which may convey other information. Like in humans, neural mechanisms underlying the exhibit of micro-expressions may differ from those of standard facial expressions. We found that some micro-expressions could be related to attention and involved in the multisensory processing of the 'fixed attention' observed in horses' high attentional state. The micro-expressions could be used by horses as social information in an interspecies relationship. We hypothesize that facial micro-expressions could be a window on transient internal states of the animal and may provide subtle and discreet social signals.

Effects of forward mask duration variability on the temporal dynamics of brief facial expression categorization

The Japanese and Caucasian Brief Affect Recognition Task (JACBART) has been proposed as a standardized method for measuring people's ability to accurately categorize briefly presented images of facial expressions. However, the factors that impact performance in this task are not entirely understood. The current study sought to explore the role of the forward mask's duration (i.e., fixed vs. variable) in brief affect categorization across expressions of the six basic emotions (i.e., anger, disgust, fear, happiness, sadness, and surprise) and three presentation times (i.e., 17, 67, and 500 ms). Current findings do not demonstrate evidence that a variable duration forward mask negatively impacts brief affect categorization. However, efficiency and necessity thresholds were observed to vary across the expressions of emotion. Further exploration of the temporal dynamics of facial affect categorization will therefore require a consideration of these differences.

CAS(ME)3: A Third Generation Facial Spontaneous Micro-Expression Database With Depth Information and High Ecological Validity

Micro-expression (ME) is a significant non-verbal communication clue that reveals one person's genuine emotional state. The development of micro-expression analysis (MEA) has just gained attention in the last decade. However, the small sample size problem constrains the use of deep learning on MEA. Besides, ME samples distribute in six different databases, leading to database bias. Moreover, the ME database development is complicated. In this article, we introduce a large-scale spontaneous ME database: CAS(ME) 3. The contribution of this article is summarized as follows: (1) CAS(ME) 3 offers around 80 hours of videos with over 8,000,000 frames, including manually labeled 1,109 MEs and 3,490 macro-expressions. Such a large sample size allows effective MEA method validation while avoiding database bias. (2) Inspired by psychological experiments, CAS(ME) 3 provides the depth information as an additional modality unprecedentedly, contributing to multi-modal MEA. (3) For the first time, CAS(ME) 3 elicits ME with high ecological validity using the mock crime paradigm, along with physiological and voice signals, contributing to practical MEA. (4) Besides, CAS(ME) 3 provides 1,508 unlabeled videos with more than 4,000,000 frames, i.e., a data platform for unsupervised MEA methods. (5) Finally, we demonstrate the effectiveness of depth information by the proposed depth flow algorithm and RGB-D information.

The effect of facial attractiveness on micro-expression recognition

Micro-expression (ME) is an extremely quick and uncontrollable facial movement that lasts for 40–200 ms and reveals thoughts and feelings that an individual attempts to cover up. Though much more difficult to detect and recognize, ME recognition is similar to macro-expression recognition in that it is influenced by facial features. Previous studies suggested that facial attractiveness could influence facial expression recognition processing. However, it remains unclear whether facial attractiveness could also influence ME recognition. Addressing this issue, this study tested 38 participants with two ME recognition tasks in a static condition or dynamically. Three different MEs (positive, neutral, and negative) at two attractiveness levels (attractive, unattractive). The results showed that participants recognized MEs on attractive faces much quicker than on unattractive ones, and there was a significant interaction between ME and facial attractiveness. Furthermore, attractive happy faces were recognized faster in both the static and the dynamic conditions, highlighting the happiness superiority effect. Therefore, our results provided the first evidence that facial attractiveness could influence ME recognition in a static condition or dynamically.

Oxytocin Impairs the Recognition of Micro-Expressions of Surprise and Disgust

As fleeting facial expressions which reveal the emotion that a person tries to conceal, micro-expressions have great application potentials for fields like security, national defense and medical treatment. However, the physiological basis for the recognition of these facial expressions is poorly understood. In the present research, we utilized a double-blind, placebo-controlled, mixed-model experimental design to investigate the effects of oxytocin on the recognition of micro-expressions in three behavioral studies. Specifically, in Studies 1 and 2, participants were asked to perform a laboratory-based standardized micro-expression recognition task after self-administration of a single dose of intranasal oxytocin (40 IU) or placebo (containing all ingredients except for the neuropeptide). In Study 3, we further examined the effects of oxytocin on the recognition of natural micro-expressions. The results showed that intranasal oxytocin decreased the recognition speed for standardized intense micro-expressions of surprise (Study 1) and decreased the recognition accuracy for standardized subtle micro-expressions of disgust (Study 2). The results of Study 3 further revealed that intranasal oxytocin administration significantly reduced the recognition accuracy for natural micro-expressions of surprise and disgust. The present research is the first to investigate the effects of oxytocin on micro-expression recognition. It suggests that the oxytocin mainly plays an inhibiting role in the recognition of micro-expressions and there are fundamental differences in the neurophysiological basis for the recognition of micro-expressions and macro-expressions.

The relevant resting-state brain activity of ecological microexpression recognition test (EMERT)

Zhang, et al. (2017) established the ecological microexpression recognition test (EMERT), but it only used white models’ expressions as microexpressions and backgrounds, and there was no research detecting its relevant brain activity. The current study used white, black and yellow models’ expressions as microexpressions and backgrounds to improve the materials ecological validity of EMERT, and it used eyes-closed and eyes-open resting-state fMRI to detect relevant brain activity of EMERT for the first time. The results showed: (1) Two new recapitulative indexes of EMERT were adopted, such as microexpression M and microexpression SD. The participants could effectively identify almost all the microexpressions, and each microexpression type had a significantly background effect. The EMERT had good retest reliability and calibration validity. (2) ALFFs (Amplitude of Low-Frequency Fluctuations) in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression M. The relevant brain areas of microexpression M were some frontal lobes, insula, cingulate cortex, hippocampus, parietal lobe, caudate nucleus, thalamus, amygdala, occipital lobe, fusiform, temporal lobe, cerebellum and vermis. (3) ALFFs in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression SD, and the ALFFs-difference was more predictive. The relevant brain areas of microexpression SD were some frontal lobes, insula, cingulate cortex, cuneus, amygdala, fusiform, occipital lobe, parietal lobe, precuneus, caudate lobe, putamen lobe, thalamus, temporal lobe, cerebellum and vermis. (4) There were many similarities and some differences in the relevant brain areas between microexpression M and SD. All these brain areas can be trained to enhance ecological microexpression recognition ability.

The Establishment of Pseudorandom Ecological Microexpression Recognition Test (PREMERT) and Its Relevant Resting-State Brain Activity

The EMERT (ecological microexpression recognition test) by Zhang et al. (2017) used between-subjects Latin square block design for backgrounds; therefore, participants could not get comparable scores. The current study used within-subject pseudorandom design for backgrounds to improve EMERT to PREMERT (pseudorandom EMERT) and used eyes-closed and eyes-open resting-state functional magnetic resonance imaging to detect relevant brain activity of PREMERT for the first time. The results showed (1) two new recapitulative indexes of PREMERT were adopted, such as microexpression M and microexpression SD. Using pseudorandom design, the participants could effectively identify almost all the microexpressions, and each microexpression type had significant background effect. The PREMERT had good split-half reliability, parallel-forms reliability, criterion validity, and ecological validity. Therefore, it could stably and efficiently detect the participants' microexpression recognition abilities. Because of its pseudorandom design, all participants did the same test; their scores could be compared with each other. (2) amplitude of low-frequency fluctuations (ALFF; 0.01-0.1 Hz) in both eyes-closed and eyes-open resting states and ALFF difference could predict microexpression M, and the ALFF difference was less predictive. The relevant resting-state brain areas of microexpression M were some frontal lobes, insula, cingulate cortex, hippocampus, amygdala, fusiform gyrus, parietal lobe, caudate nucleus, precuneus, thalamus, putamen, temporal lobe, and cerebellum. (3) ALFFs in both eyes-closed and eyes-open resting states and ALFF difference could predict microexpression SD, and the ALFF difference was more predictive. The relevant resting-state brain areas of microexpression SD were some frontal lobes, central anterior gyrus, supplementary motor area, insula, hippocampus, amygdala, cuneus, occipital lobe, fusiform gyrus, parietal lobe, caudate nucleus, pallidum, putamen, thalamus, temporal lobe, and cerebellum. (4) There were many similar relevant resting-state brain areas, such as brain areas of expression recognition, microexpressions consciousness and attention, and the change from expression backgrounds to microexpression, and some different relevant resting-state brain areas, such as precuneus, insula, and pallidum, between microexpression M and SD. The ALFF difference was more sensitive to PREMERT fluctuations.

Brain Activation in Contrasts of Microexpression Following Emotional Contexts

The recognition of microexpressions may be influenced by emotional contexts. The microexpression is recognized poorly when it follows a negative context in contrast to a neutral context. Based on the behavioral evidence, we predicted that the effect of emotional contexts might be dependent on neural activities. Using the synthesized microexpressions task modified from the Micro-Expression Training Tool (METT), we performed an functional MRI (fMRI) study to compare brain response in contrasts of the same targets following different contexts. Behaviorally, we observed that the accuracies of target microexpressions following neutral contexts were significantly higher than those following negative or positive contexts. At the neural level, we found increased brain activations in contrasts of the same targets following different contexts, which reflected the discrepancy in the processing of emotional contexts. The increased activations implied that different emotional contexts might differently influence the processing of subsequent target microexpressions and further suggested interactions between the processing of emotional contexts and of microexpressions.

Ecological micro-expression recognition characteristics of young adults with subthreshold depression

The micro-expression (ME) processing characteristics of patients with depression has been studied but has not been investigated in people with subthreshold depression. Based on this, by adopting the ecological MEs recognition paradigm, this study aimed to explore ME recognition in people with subthreshold depression. A 4 (background expression: happy, neutral, sad and fearful) × 4 (ME: happy, neutral, sad, and fearful) study was designed; two groups of participants (experimental group with subthreshold depression vs. healthy control group, 32 participants in each group) were asked to complete the ecological ME recognition task, and the corresponding accuracy (ACC) and reaction time (RT) were analyzed. Results: (1) Under different background conditions, recognizing happy MEs had the highest ACC and shortest RT. (2) There was no significant difference in the ACC and RT between experimental and control groups. (3)In different contexts, individuals with subthreshold depression tended to misjudge neutral, sad, and fearful MEs as happy, while neutral MEs were misjudged as sad and fearful. (4) The performance of individuals with subthreshold depression in the ecological ME recognition task were influenced by the type of ME; they showed highest ACC and shortest RT when recognizing happy MEs (vs. the other MEs). Conclusions: (1) The performance of individuals’ ecological ME recognition were influenced by the background expression, and this embodied the need for ecological ME recognition. (2) Individuals with subthreshold depression showed normal ecological ME recognition ability. (3) In terms of misjudgment, individuals with subthreshold depression showed both positive and negative bias, when completing the ecological ME recognition task. (4) Compared with the other MEs, happy MEs showed an advantage recognition effect for individuals with subthreshold depression who completed the ecological ME recognition task.

The Establishment of Weak Ecological Microexpressions Recognition Test (WEMERT): An Extension on EMERT

The JACBART (Japanese and Caucasian Brief Affect Recognition Test) microexpression recognition test only examines facial expressions under the neutral expression background and the ecological validity is not high. The EMERT (Ecological MicroExpressions Recognition Test) microexpression recognition test examined six microexpressions under seven backgrounds but does not detect the intensity of expressions. In the current study, a weak ecological microexpression recognition test was established to examine the recognition features of six weak microexpressions in all seven high intensity basic expressions. The results found: (1) the test had good retest reliability, criterion validity and ecological validity; and (2) the reliability and validity tests revealed a lot of characteristics of weak microexpression recognition. There were training effects in some weak microexpression recognition. Weak microexpression recognition was generally positively related to the microexpression recognition of JACBART but were generally negatively related to approximate common expressions. The backgrounds main effects in all weak microexpressions were significant and pairwise comparisons show there were a wide range of differences between weak microexpressions under different backgrounds. The standard deviations, of the accuracy of weak microexpressions in different backgrounds, were used to define the fluctuations of the weak microexpression recognition and we found that weak microexpression recognition had many fluctuations. (3) Personality openness and its subdimensions (O1, O2, O3, and O5) were generally positively related to some weak microexpression recognition, except O1, which was significantly negatively related to surprise under neutrality. O1 was positively related to the standard deviation of the weak anger microexpression recognition accuracies and O6 was negatively related to the standard deviation of the weak happiness microexpression recognition accuracies in the first measurement.

Emotional expression in psychiatric conditions: New technology for clinicians

AIM

Emotional expressions are one of the most widely studied topics in neuroscience, from both clinical and non-clinical perspectives. Atypical emotional expressions are seen in various psychiatric conditions, including schizophrenia, depression, and autism spectrum conditions. Understanding the basics of emotional expressions and recognition can be crucial for diagnostic and therapeutic procedures. Emotions can be expressed in the face, gesture, posture, voice, and behavior and affect physiological parameters, such as the heart rate or body temperature. With modern technology, clinicians can use a variety of tools ranging from sophisticated laboratory equipment to smartphones and web cameras. The aim of this paper is to review the currently used tools using modern technology and discuss their usefulness as well as possible future directions in emotional expression research and treatment strategies.

METHODS

The authors conducted a literature review in the PubMed, EBSCO, and SCOPUS databases, using the following key words: 'emotions,' 'emotional expression,' 'affective computing,' and 'autism.' The most relevant and up-to-date publications were identified and discussed. Search results were supplemented by the authors' own research in the field of emotional expression.

RESULTS

We present a critical review of the currently available technical diagnostic and therapeutic methods. The most important studies are summarized in a table.

CONCLUSION

Most of the currently available methods have not been adequately validated in clinical settings. They may be a great help in everyday practice; however, they need further testing. Future directions in this field include more virtual-reality-based and interactive interventions, as well as development and improvement of humanoid robots.

A False Trail to Follow: Differential Effects of the Facial Feedback Signals From the Upper and Lower Face on the Recognition of Micro-Expressions

Micro-expressions, as fleeting facial expressions, are very important for judging people’s true emotions, thus can provide an essential behavioral clue for lie and dangerous demeanor detection. From embodied accounts of cognition, we derived a novel hypothesis that facial feedback from upper and lower facial regions has differential effects on micro-expression recognition. This hypothesis was tested and supported across three studies. Specifically, the results of Study 1 showed that people became better judges of intense micro-expressions with a duration of 450 ms when the facial feedback from upper face was enhanced via a restricting gel. Additional results of Study 2 showed that the recognition accuracy of subtle micro-expressions was significantly impaired under all duration conditions (50, 150, 333, and 450 ms) when facial feedback from lower face was enhanced. In addition, the results of Study 3 also revealed that blocking the facial feedback of lower face, significantly boosted the recognition accuracy of subtle and intense micro-expressions under all duration conditions (150 and 450 ms). Together, these results highlight the role of facial feedback in judging the subtle movements of micro-expressions.

Effects of task-irrelevant emotional information on deception

Deception has been reported to be influenced by task-relevant emotional information from an external stimulus. However, it remains unclear how task-irrelevant emotional information would influence deception. In the present study, facial expressions of different valence and emotion intensity were presented to participants, where they were asked to make either truthful or deceptive gender judgments according to the preceding cues. We observed the influence of facial expression intensity upon individuals' cognitive cost of deceiving (mean difference of individuals' truthful and deceptive response times). Larger cost was observed for high intensity faces compared to low intensity faces. These results provided insights on how automatic attraction of attention evoked by task-irrelevant emotional information in facial expressions influenced individuals' cognitive cost of deceiving.

Comparison of Ecological Micro-Expression Recognition in Patients with Depression and Healthy Individuals

Previous studies have focused on the characteristics of ordinary facial expressions in patients with depression, and have not investigated the processing characteristics of ecological micro-expressions (MEs, i.e., MEs that presented in different background expressions) in these patients. Based on this, adopting the ecological MEs recognition paradigm, this study aimed to comparatively evaluate facial ME recognition in depressed and healthy individuals. The findings of the study are as follows: (1) background expression: the accuracy (ACC) in the neutral background condition tended to be higher than that in the fear background condition, and the reaction time (RT) in the neutral background condition was significantly longer than that in other backgrounds. The type of ME and its interaction with the type of background expression could affect participants’ ecological MEs recognition ACC and speed. Depression type: there was no significant difference between the ecological MEs recognition ACC of patients with depression and healthy individuals, but the patients’ RT was significantly longer than that of healthy individuals; and (2) patients with depression judged happy MEs that were presented against different backgrounds as neutral and judged neutral MEs that were presented against sad backgrounds as sad. The present study suggested the following: (1) ecological MEs recognition was influenced by background expressions. The ACC of happy MEs was the highest, of neutral ME moderate and of sadness and fear the lowest. The response to the happy MEs was significantly shorter than that of identifying other MEs. It is necessary to conduct research on ecological MEs recognition; (2) the speed of patients with depression in identifying ecological MEs was slower than of healthy individuals; indicating that the patients’ cognitive function was impaired; and (3) the patients with depression showed negative bias in the ecological MEs recognition task, reflecting the lack of happy ME recognition ability and the generalized identification of sad MEs in those patients.

The Auditory Kuleshov Effect: Multisensory Integration in Movie Editing

Almost a hundred years ago, the Russian filmmaker Lev Kuleshov conducted his now famous editing experiment in which different objects were added to a given film scene featuring a neutral face. It is said that the audience interpreted the unchanged facial expression as a function of the added object (e.g., an added soup made the face express hunger). This interaction effect has been dubbed "Kuleshov effect." In the current study, we explored the role of sound in the evaluation of facial expressions in films. Thirty participants watched different clips of faces that were intercut with neutral scenes, featuring either happy music, sad music, or no music at all. This was crossed with the facial expressions of happy, sad, or neutral. We found that the music significantly influenced participants' emotional judgments of facial expression. Thus, the intersensory effects of music are more specific than previously thought. They alter the evaluation of film scenes and can give meaning to ambiguous situations.

Effects of micro- and subtle-expression reading skill training in medical students: A randomized trial

OBJECTIVE

to investigate the effectiveness of the Micro Expression Training Tool (METT) and the Subtle Expression Training Tool (SETT) to help improve the non-verbal communication skills of medical students.

METHODS

In a randomized controlled trial, all participants were randomly allocated to either a training (n=41) or control group (n=41) and were pre-tested before education with METT and SETT at baseline. Then, training students took second tests after a 1-h class about interpreting micro and subtle expressions and control students took the second tests without the class.

RESULTS

METT pre-test scores were positively related with female gender, agreeableness, whereas SETT pre-test scores were negatively related with age and positively related with female gender. Mean METT score increases of 29.3% and mean SETT score increases of 36.2% were observed after training, whereas the control group achieved only a mean METT score increase of 11.0% at second testing. Increases in both test scores in the training group were significantly higher than in the control group.

CONCLUSION

METT and SETT are effective, simple tools for improving the micro- and subtle-expression reading skills of medical students.

PRACTICE IMPLICATIONS

METT and SETT can be effective for improving the non-verbal communication skills of medical students.