Tracking the affective state of unseen persons

Development of the RIKEN database for dynamic facial expressions with multiple angles

The development of facial expressions with sensing information is progressing in multidisciplinary fields, such as psychology, affective computing, and cognitive science. Previous facial datasets have not simultaneously dealt with multiple theoretical views of emotion, individualized context, or multi-angle/depth information. We developed a new facial database (RIKEN facial expression database) that includes multiple theoretical views of emotions and expressers' individualized events with multi-angle and depth information. The RIKEN facial expression database contains recordings of 48 Japanese participants captured using ten Kinect cameras at 25 events. This study identified several valence-related facial patterns and found them consistent with previous research investigating the coherence between facial movements and internal states. This database represents an advancement in developing a new sensing system, conducting psychological experiments, and understanding the complexity of emotional events.

Multi-view emotional expressions dataset using 2D pose estimation

Human body expressions convey emotional shifts and intentions of action and, in some cases, are even more effective than other emotion models. Despite many datasets of body expressions incorporating motion capture available, there is a lack of more widely distributed datasets regarding naturalized body expressions based on the 2D video. In this paper, therefore, we report the multi-view emotional expressions dataset (MEED) using 2D pose estimation. Twenty-two actors presented six emotional (anger, disgust, fear, happiness, sadness, surprise) and neutral body movements from three viewpoints (left, front, right). A total of 4102 videos were captured. The MEED consists of the corresponding pose estimation results (i.e., 397,809 PNG files and 397,809 JSON files). The size of MEED exceeds 150 GB. We believe this dataset will benefit the research in various fields, including affective computing, human-computer interaction, social neuroscience, and psychiatry.

Emotion prediction as computation over a generative theory of mind

From sparse descriptions of events, observers can make systematic and nuanced predictions of what emotions the people involved will experience. We propose a formal model of emotion prediction in the context of a public high-stakes social dilemma. This model uses inverse planning to infer a person's beliefs and preferences, including social preferences for equity and for maintaining a good reputation. The model then combines these inferred mental contents with the event to compute 'appraisals': whether the situation conformed to the expectations and fulfilled the preferences. We learn functions mapping computed appraisals to emotion labels, allowing the model to match human observers' quantitative predictions of 20 emotions, including joy, relief, guilt and envy. Model comparison indicates that inferred monetary preferences are not sufficient to explain observers' emotion predictions; inferred social preferences are factored into predictions for nearly every emotion. Human observers and the model both use minimal individualizing information to adjust predictions of how different people will respond to the same event. Thus, our framework integrates inverse planning, event appraisals and emotion concepts in a single computational model to reverse-engineer people's intuitive theory of emotions. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

Inferential Emotion Tracking reveals impaired context-based emotion processing in individuals with high Autism Quotient scores

Emotion perception is essential for successful social interactions and maintaining long-term relationships with friends and family. Individuals with autism spectrum disorder (ASD) experience social communication deficits and have reported difficulties in facial expression recognition. However, emotion recognition depends on more than just processing face expression; context is critically important to correctly infer the emotions of others. Whether context-based emotion processing is impacted in those with Autism remains unclear. Here, we used a recently developed context-based emotion perception task, called Inferential Emotion Tracking (IET), and investigated whether individuals who scored high on the Autism Spectrum Quotient (AQ) had deficits in context-based emotion perception. Using 34 videos (including Hollywood movies, home videos, and documentaries), we tested 102 participants as they continuously tracked the affect (valence and arousal) of a blurred-out, invisible character. We found that individual differences in Autism Quotient scores were more strongly correlated with IET task accuracy than they are with traditional face emotion perception tasks. This correlation remained significant even when controlling for potential covarying factors, general intelligence, and performance on traditional face perception tasks. These findings suggest that individuals with ASD may have impaired perception of contextual information, it reveals the importance of developing ecologically relevant emotion perception tasks in order to better assess and treat ASD, and it provides a new direction for further research on context-based emotion perception deficits in ASD.

Facial expression and emotion

Human facial expressions are unique in their ability to express our emotions and communicate them to others. The mimic expression of basic emotions is very similar across different cultures and has also many features in common with other mammals. This suggests a common genetic origin of the association between facial expressions and emotion. However, recent studies also show cultural influences and differences. The recognition of emotions from facial expressions, as well as the process of expressing one's emotions facially, occurs within an extremely complex cerebral network. Due to the complexity of the cerebral processing system, there are a variety of neurological and psychiatric disorders that can significantly disrupt the coupling of facial expressions and emotions. Wearing masks also limits our ability to convey and recognize emotions through facial expressions. Through facial expressions, however, not only "real" emotions can be expressed, but also acted ones. Thus, facial expressions open up the possibility of faking socially desired expressions and also of consciously faking emotions. However, these pretenses are mostly imperfect and can be accompanied by short-term facial movements that indicate the emotions that are actually present (microexpressions). These microexpressions are of very short duration and often barely perceptible by humans, but they are the ideal application area for computer-aided analysis. This automatic identification of microexpressions has not only received scientific attention in recent years, but its use is also being tested in security-related areas. This article summarizes the current state of knowledge of facial expressions and emotions.

Contextualising facial expressions: The effect of temporal context and individual differences on classification

The influence of context on facial expression classification is most often investigated using simple cues in static faces portraying basic expressions with a fixed emotional intensity. We examined (1) whether a perceptually rich, dynamic audiovisual context, presented in the form of movie clips (to achieve closer resemblance to real life), affected the subsequent classification of dynamic basic (happy) and non-basic (sarcastic) facial expressions and (2) whether people's susceptibility to contextual cues was related to their ability to classify facial expressions viewed in isolation. Participants classified facial expressions-gradually progressing from neutral to happy/sarcastic in increasing intensity-that followed movie clips. Classification was relatively more accurate and faster when the preceding context predicted the upcoming expression, compared with when the context did not. Speeded classifications suggested that predictive contexts reduced the emotional intensity required to be accurately classified. More importantly, we show for the first time that participants' accuracy in classifying expressions without an informative context correlated with the magnitude of the contextual effects experienced by them-poor classifiers of isolated expressions were more susceptible to a predictive context. Our findings support the emerging view that contextual cues and individual differences must be considered when explaining mechanisms underlying facial expression classification.

The neural correlates of context driven changes in the emotional response: An fMRI study

Emotional flexibility reflects the ability to adjust the emotional response to the changing environmental context. To understand how context can trigger a change in emotional response, i.e., how it can upregulate the initial emotional response or trigger a shift in the valence of emotional response, we used a task consisting of picture pairs during functional magnetic resonance imaging sessions. In each pair, the first picture was a smaller detail (a decontextualized photograph depicting emotions using primarily facial and postural expressions) from the second (contextualized) picture, and the neural response to a decontextualized picture was compared with the same picture in a context. Thirty-one healthy participants (18 females; mean age: 24.44 ± 3.4) were involved in the study. In general, context (vs. pictures without context) increased activation in areas involved in facial emotional processing (e.g., middle temporal gyrus, fusiform gyrus, and temporal pole) and affective mentalizing (e.g., precuneus, temporoparietal junction). After excluding the general effect of context by using an exclusive mask with activation to context vs. no-context, the automatic shift from positive to negative valence induced by the context was associated with increased activation in the thalamus, caudate, medial frontal gyrus and lateral orbitofrontal cortex. When the meaning changed from negative to positive, it resulted in a less widespread activation pattern, mainly in the precuneus, middle temporal gyrus, and occipital lobe. Providing context cues to facial information recruited brain areas that induced changes in the emotional responses and interpretation of the emotional situations automatically to support emotional flexibility.

The Effects of Separate Facial Areas on Emotion Recognition in Different Adult Age Groups: A Laboratory and a Naturalistic Study

The identification of facial expressions is critical for social interaction. The ability to recognize facial emotional expressions declines with age. These age effects have been associated with differential age-related looking patterns. The present research project set out to systematically test the role of specific facial areas for emotion recognition across the adult lifespan. Study 1 investigated the impact of displaying only separate facial areas versus the full face on emotion recognition in 62 younger (20–24 years) and 65 middle-aged adults (40–65 years). Study 2 examined if wearing face masks differentially compromises younger (18–33 years, N = 71) versus middle-aged to older adults’ (51–83 years, N = 73) ability to identify different emotional expressions. Results of Study 1 suggested no general decrease in emotion recognition across the lifespan; instead, age-related performance seems to depend on the specific emotion and presented face area. Similarly, Study 2 observed only deficits in the identification of angry, fearful, and neutral expressions in older adults, but no age-related differences with regards to happy, sad, and disgusted expressions. Overall, face masks reduced participants’ emotion recognition; however, there were no differential age effects. Results are discussed in light of current models of age-related changes in emotion recognition.

Commercial Use of Emotion Artificial Intelligence (AI): Implications for Psychiatry

PURPOSE OF REVIEW

Emotion artificial intelligence (AI) is technology for emotion detection and recognition. Emotion AI is expanding rapidly in commercial and government settings outside of medicine, and will increasingly become a routine part of daily life. The goal of this narrative review is to increase awareness both of the widespread use of emotion AI, and of the concerns with commercial use of emotion AI in relation to people with mental illness.

RECENT FINDINGS

This paper discusses emotion AI fundamentals, a general overview of commercial emotion AI outside of medicine, and examples of the use of emotion AI in employee hiring and workplace monitoring. The successful re-integration of patients with mental illness into society must recognize the increasing commercial use of emotion AI. There are concerns that commercial use of emotion AI will increase stigma and discrimination, and have negative consequences in daily life for people with mental illness. Commercial emotion AI algorithm predictions about mental illness should not be treated as medical fact.

Diagnosis of Depressive Disorder Model on Facial Expression Based on Fast R-CNN

This study examines related literature to propose a model based on artificial intelligence (AI), that can assist in the diagnosis of depressive disorder. Depressive disorder can be diagnosed through a self-report questionnaire, but it is necessary to check the mood and confirm the consistency of subjective and objective descriptions. Smartphone-based assistance in diagnosing depressive disorders can quickly lead to their identification and provide data for intervention provision. Through fast region-based convolutional neural networks (R-CNN), a deep learning method that recognizes vector-based information, a model to assist in the diagnosis of depressive disorder can be devised by checking the position change of the eyes and lips, and guessing emotions based on accumulated photos of the participants who will repeatedly participate in the diagnosis of depressive disorder.

The Unique and Interactive Effects of Faces, Postures, and Scenes on Emotion Categorization

There is ongoing debate as to whether emotion perception is determined by facial expressions or context (i.e., non-facial cues). The present investigation examined the independent and interactive effects of six emotions (anger, disgust, fear, joy, sadness, neutral) conveyed by combinations of facial expressions, bodily postures, and background scenes in a fully crossed design. Participants viewed each face-posture-scene (FPS) combination for 5 s and were then asked to categorize the emotion depicted in the image. Four key findings emerged from the analyses: (1) For fully incongruent FPS combinations, participants categorized images using the face in 61% of instances and the posture and scene in 18% and 11% of instances, respectively; (2) postures (with neutral scenes) and scenes (with neutral postures) exerted differential influences on emotion categorizations when combined with incongruent facial expressions; (3) contextual asymmetries were observed for some incongruent face-posture pairings and their inverse (e.g., anger-fear vs. fear-anger), but not for face-scene pairings; (4) finally, scenes exhibited a boosting effect of posture when combined with a congruent posture and attenuated the effect of posture when combined with a congruent face. Overall, these findings highlight independent and interactional roles of posture and scene in emotion face perception. Theoretical implications for the study of emotions in context are discussed.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-021-00061-x.

Searching for a paradigm shift in the research on the epilepsies and associated neuropsychiatric comorbidities. From ancient historical knowledge to the challenge of contemporary systems complexity and emergent functions

In this review, we will discuss in four scenarios our challenges to offer possible solutions for the puzzle associated with the epilepsies and neuropsychiatric comorbidities. We need to recognize that (1) since quite old times, human wisdom was linked to the plural (distinct global places/cultures) perception of the Universe we are in, with deep respect for earth and nature. Plural ancestral knowledge was added with the scientific methods; however, their joint efforts are the ideal scenario; (2) human behavior is not different than animal behavior, in essence the product of Darwinian natural selection; knowledge of animal and human behavior are complementary; (3) the expression of human behavior follows the same rules that complex systems with emergent properties, therefore, we can measure events in human, clinical, neurobiological situations with complexity systems' tools; (4) we can use the semiology of epilepsies and comorbidities, their neural substrates, and potential treatments (including experimental/computational modeling, neurosurgical interventions), as a source and collection of integrated big data to predict with them (e.g.: machine/deep learning) diagnosis/prognosis, individualized solutions (precision medicine), basic underlying mechanisms and molecular targets. Once the group of symptoms/signals (with a myriad of changing definitions and interpretations over time) and their specific sequences are determined, in epileptology research and clinical settings, the use of modern and contemporary techniques such as neuroanatomical maps, surface electroencephalogram and stereoelectroencephalography (SEEG) and imaging (MRI, BOLD, DTI, SPECT/PET), neuropsychological testing, among others, are auxiliary in the determination of the best electroclinical hypothesis, and help design a specific treatment, usually as the first attempt, with available pharmacological resources. On top of ancient knowledge, currently known and potentially new antiepileptic drugs, alternative treatments and mechanisms are usually produced as a consequence of the hard, multidisciplinary, and integrated studies of clinicians, surgeons, and basic scientists, all over the world. The existence of pharmacoresistant patients, calls for search of other solutions, being along the decades the surgeries the most common interventions, such as resective procedures (i.e., selective or standard lobectomy, lesionectomy), callosotomy, hemispherectomy and hemispherotomy, added by vagus nerve stimulation (VNS), deep brain stimulation (DBS), neuromodulation, and more recently focal minimal or noninvasive ablation. What is critical when we consider the pharmacoresistance aspect with the potential solution through surgery, is still the pursuit of localization-dependent regions (e.g.: epileptogenic zone (EZ)), in order to decide, no matter how sophisticated are the brain mapping tools (EEG and MRI), the size and location of the tissue to be removed. Mimicking the semiology and studying potential neural mechanisms and molecular targets - by means of experimental and computational modeling - are fundamental steps of the whole process. Concluding, with the conjunction of ancient knowledge, coupled to critical and creative contemporary, scientific (not dogmatic) clinical/surgical, and experimental/computational contributions, a better world and of improved quality of life can be offered to the people with epilepsy and neuropsychiatric comorbidities, who are still waiting (as well as the scientists) for a paradigm shift in epileptology, both in the Basic Science, Computational, Clinical, and Neurosurgical Arenas. This article is part of the Special Issue "NEWroscience 2018".

Inferential affective tracking reveals the remarkable speed of context-based emotion perception

Understanding the emotional states of others is important for social functioning. Recent studies show that context plays an essential role in emotion recognition. However, it remains unclear whether emotion inference from visual scene context is as efficient as emotion recognition from faces. Here, we measured the speed of context-based emotion perception, using Inferential Affective Tracking (IAT) with naturalistic and dynamic videos. Using cross-correlation analyses, we found that inferring affect based on visual context alone is just as fast as tracking affect with all available information including face and body. We further demonstrated that this approach has high precision and sensitivity to sub-second lags. Our results suggest that emotion recognition from dynamic contextual information might be automatic and immediate. Seemingly complex context-based emotion perception is far more efficient than previously assumed.

Social context and culture influence judgments of non-Duchenne smiles

Extant evidence points toward the role of contextual information and related cross-cultural variations in emotion perception, but most of the work to date has focused on judgments of basic emotions. The current research examines how culture and situational context affect the interpretation of emotion displays, i.e. judgments of the extent to which ambiguous smiles communicate happiness versus polite intentions. We hypothesized that smiles associated with contexts implying happiness would be judged as conveying more positive feelings compared to smiles paired with contexts implying politeness or smiles presented without context. In line with existing research on cross-cultural variation in contextual influences, we also expected these effects to be larger in Japan than in the UK. In Study 1, British participants viewed non-Duchenne smiles presented on their own or paired with background scenes implying happiness or the need to be polite. Compared to face-only stimuli, happy contexts made smiles appear more genuine, whereas polite contexts led smiles to be seen as less genuine. Study 2 replicated this result using verbal vignettes, showing a similar pattern of contextual effects among British and Japanese participants. However, while the effects of vignettes describing happy situations was comparable in both cultures, the influence of vignettes describing polite situations was stronger in Japan than the UK. Together, the findings document the importance of context information in judging smile expressions and highlight the need to investigate how culture moderates such influences.

Kinematic dataset of actors expressing emotions

Human body movements can convey a variety of emotions and even create advantages in some special life situations. However, how emotion is encoded in body movements has remained unclear. One reason is that there is a lack of public human body kinematic dataset regarding the expressing of various emotions. Therefore, we aimed to produce a comprehensive dataset to assist in recognizing cues from all parts of the body that indicate six basic emotions (happiness, sadness, anger, fear, disgust, surprise) and neutral expression. The present dataset was created using a portable wireless motion capture system. Twenty-two semi-professional actors (half male) completed performances according to the standardized guidance and preferred daily events. A total of 1402 recordings at 125 Hz were collected, consisting of the position and rotation data of 72 anatomical nodes. To our knowledge, this is now the largest emotional kinematic dataset of the human body. We hope this dataset will contribute to multiple fields of research and practice, including social neuroscience, psychiatry, computer vision, and biometric and information forensics.

Emotional judgments of scenes are influenced by unintentional averaging

Background

The visual system uses ensemble perception to summarize visual input across a variety of domains. This heuristic operates at multiple levels of vision, compressing information as basic as oriented lines or as complex as emotional faces. Given its pervasiveness, the ensemble unsurprisingly can influence how an individual item is perceived, and vice versa.

Methods

In the current experiments, we tested whether the perceived emotional valence of a single scene could be influenced by surrounding, simultaneously presented scenes. Observers first rated the emotional valence of a series of individual scenes. They then saw ensembles of the original images, presented in sets of four, and were cued to rate, for a second time, one of four.

Results

Results confirmed that the perceived emotional valence of the cued image was pulled toward the mean emotion of the surrounding ensemble on the majority of trials, even though the ensemble was task-irrelevant. Control experiments and analyses confirmed that the pull was driven by high-level, ensemble information.

Conclusion

We conclude that high-level ensemble information can influence how we perceive individual items in a crowd, even when working memory demands are low and the ensemble information is not directly task-relevant.

The bottom-up and top-down processing of faces in the human occipitotemporal cortex

Although face processing has been studied extensively, the dynamics of how face-selective cortical areas are engaged remains unclear. Here, we uncovered the timing of activation in core face-selective regions using functional Magnetic Resonance Imaging and Magnetoencephalography in humans. Processing of normal faces started in the posterior occipital areas and then proceeded to anterior regions. This bottom-up processing sequence was also observed even when internal facial features were misarranged. However, processing of two-tone Mooney faces lacking explicit prototypical facial features engaged top-down projection from the right posterior fusiform face area to right occipital face area. Further, face-specific responses elicited by contextual cues alone emerged simultaneously in the right ventral face-selective regions, suggesting parallel contextual facilitation. Together, our findings chronicle the precise timing of bottom-up, top-down, as well as context-facilitated processing sequences in the occipital-temporal face network, highlighting the importance of the top-down operations especially when faced with incomplete or ambiguous input.

Embodied Cognition in Performance: The Impact of Michael Chekhov's Acting Exercises on Affect and Height Perception

Modern embodied approaches to cognitive science overlap with ideas long explored in theater. Performance coaches such as Michael Chekhov have emphasized proprioceptive awareness of movement as a path to attaining psychological states relevant for embodying characters and inhabiting fictional spaces. Yet, the psychology of performance remains scientifically understudied. Experiments, presented in this paper, investigated the effects of three sets of exercises adapted from Chekhov's influential techniques for actors' training. Following a continuous physical demonstration and verbal prompts by the actress Bonnie Eckard, 29 participants enacted neutral, expanding, and contracting gestures and attitudes in space. After each set of exercises, the participants' affect (pleasantness and arousal) and self-perceptions of height were measured. Within the limitations of the study, we measured a significant impact of the exercises on affect: pleasantness increased by 50% after 15 min of expanding exercises and arousal increased by 15% after 15 min of contracting exercises, each relative to the other exercise. Although the exercises produced statistically non-significant changes in the perceived height, there was a significant relation between perceived height and affect, in which perceived height increased with increases in either pleasantness, or arousal. These findings provide a preliminary support for Chekhov's intuition that expanding and contracting physical actions exert opposite effects on the practitioners' psychological experience. Further studies are needed to consider a wider range of factors at work in Chekhov's method and the embodied experience of acting in general.

Mapping the Passions: Toward a High-Dimensional Taxonomy of Emotional Experience and Expression

What would a comprehensive atlas of human emotions include? For 50 years, scientists have sought to map emotion-related experience, expression, physiology, and recognition in terms of the "basic six"-anger, disgust, fear, happiness, sadness, and surprise. Claims about the relationships between these six emotions and prototypical facial configurations have provided the basis for a long-standing debate over the diagnostic value of expression (for review and latest installment in this debate, see Barrett et al., p. 1). Building on recent empirical findings and methodologies, we offer an alternative conceptual and methodological approach that reveals a richer taxonomy of emotion. Dozens of distinct varieties of emotion are reliably distinguished by language, evoked in distinct circumstances, and perceived in distinct expressions of the face, body, and voice. Traditional models-both the basic six and affective-circumplex model (valence and arousal)-capture a fraction of the systematic variability in emotional response. In contrast, emotion-related responses (e.g., the smile of embarrassment, triumphant postures, sympathetic vocalizations, blends of distinct expressions) can be explained by richer models of emotion. Given these developments, we discuss why tests of a basic-six model of emotion are not tests of the diagnostic value of facial expression more generally. Determining the full extent of what facial expressions can tell us, marginally and in conjunction with other behavioral and contextual cues, will require mapping the high-dimensional, continuous space of facial, bodily, and vocal signals onto richly multifaceted experiences using large-scale statistical modeling and machine-learning methods.