Emotional expressions beyond facial muscle actions. A call for studying autonomic signals and their impact on social perception

Sensorimotor regulation of facial expression - An untouched frontier

Facial expression is a critical form of nonverbal social communication which promotes emotional exchange and affiliation among humans. Facial expressions are generated via precise contraction of the facial muscles, guided by sensory feedback. While the neural pathways underlying facial motor control are well characterized in humans and primates, it remains unknown how tactile and proprioceptive information reaches these pathways to guide facial muscle contraction. Thus, despite the importance of facial expressions for social functioning, little is known about how they are generated as a unique sensorimotor behavior. In this review, we highlight current knowledge about sensory feedback from the face and how it is distinct from other body regions. We describe connectivity between the facial sensory and motor brain systems, and call attention to the other brain systems which influence facial expression behavior, including vision, gustation, emotion, and interoception. Finally, we petition for more research on the sensory basis of facial expressions, asserting that incomplete understanding of sensorimotor mechanisms is a barrier to addressing atypical facial expressivity in clinical populations.

Neurobehavioral meaning of pupil size

Pupil size is a widely used metric of brain state. It is one of the few signals originating from the brain that can be readily monitored with low-cost devices in basic science, clinical, and home settings. It is, therefore, important to investigate and generate well-defined theories related to specific interpretations of this metric. What exactly does it tell us about the brain? Pupils constrict in response to light and dilate during darkness, but the brain also controls pupil size irrespective of luminosity. Pupil size fluctuations resulting from ongoing "brain states" are used as a metric of arousal, but what is pupil-linked arousal and how should it be interpreted in neural, cognitive, and computational terms? Here, we discuss some recent findings related to these issues. We identify open questions and propose how to answer them through a combination of well-defined tasks, neurocomputational models, and neurophysiological probing of the interconnected loops of causes and consequences of pupil size.

Rapid facial mimicry in Platyrrhini: Play face replication in spider monkeys (Ateles fusciceps, Ateles hybridus, and Ateles paniscus)

Rapid facial mimicry (RFM), the rapid and automatic replication of facial expression perceived, is considered a basic form of empathy and was investigated mainly during play. RFM occurs in Catarrhini (Old World primates), but it is not still demonstrated in Platyrrhini (New World primates). For this reason, we collected video data on playful interactions (Nplay_interactions = 149) in three species of spider monkeys (Ateles fusciceps-N = 11, Ateles hybridus-N = 14, and Ateles paniscus-N = 6) housed at La Vallée des Singes and the ZooParc de Beauval (France). For the first time, we demonstrated the occurrence of RFM in Platyrrhini (analyzing 175 events). Players' sex, age, species, relationship quality, and kinship did not modulate RFM probably due to the species' complex fission-fusion dynamics and flexible interindividual social relationships. Compared to the absence of any playful expressions or the presence of only not replicated play face, RFM prolonged the session duration and was sequentially associated with more types of more intense offensive playful patterns (patterns aimed at attacking/pursuing the playmate). We proposed that RFM may favor synchronization and context sharing between players, thus decreasing the risk of behavior misinterpretation while simultaneously fostering a more competitive nature of play. In conclusion, this study stimulates additional research on the evolutionary origins of motor mimicry in primates, possibly dating back to before the divergence of New and Old World monkeys. Furthermore, it also points toward the possibility that RFM may not always lead to cooperation but also to competition, depending on the context and species' social and cognitive features.

Reactivity of observers' facial skin blood flow depending on others' facial expressions and blushing

Facial skin blood flow (SkBF) has attracted attention as an autonomic indicator because it influences facial colour, which informs others of emotional states, and facial temperature related to social anxiety. Previous studies have examined the facial SkBF in people experiencing emotions; however, facial SkBF changes in the observers of emotions are poorly understood. Our study clarified facial SkBF changes related to observing others' emotions by comparing the changes with other physiological indices. Thirty healthy participants (24 females; mean age: 22.17) observed six types of facial expressions (neutral, angry, and embarrassed expressions with and without facial blushing) and rated the emotional intensity of the other person. We measured their facial SkBF, finger SkBF, and cardiac RR interval as they made their observations. Facial SkBF generally decreased in relation to observing emotional faces (angry and embarrassed faces) and significantly decreased for angry expressions with blushing. None of the participants noticed blushing of facial stimuli. For the RR interval and finger SkBF, there was no variation depending on the observed facial expressions, although there was a general increase related to observation. These results indicated that facial SkBF is sensitive and reactive to emotional faces-especially angry faces with blushing- compared with other autonomic indices. The facial SkBF changes were not related to either RR interval changes or the intensity rating, suggesting that facial SkBF changes may be caused by vasoconstriction and have potential functions for our emotions. The decrease in facial SkBF may have a role in calming observers by preventing them from adopting the same emotional state as a person with intense anger. These findings clarify daily facial SkBF fluctuations and their relationship with our emotional processing in interpersonal situations.

The Reward-Related Shift of Emotional Contagion from the Observer's Perspective Correlates to Their Intimacy with the Expresser

Although previous studies have found a bidirectional relationship between emotional contagion and reward, there is insufficient research to prove the effect of reward on the social function of emotional contagion. To explore this issue, the current study used electroencephalography (EEG) and the interactive way in which the expresser played games to help participants obtain reward outcomes. The results demonstrated a significant correlation between changes in emotional contagion and closeness, indicating that emotional contagion has a social regulatory function. Regarding the impact of reward outcomes, the results showed that compared to the context of a loss, in the context of a win, participants' closeness toward the expresser shifted to a more intimate level, their emotional contagion changed in a more positive direction, and the activity of the late positive component (LPC) of the event-related potentials (ERPs) changed to a greater extent. Significantly, the mediation results demonstrated the effect of reward and indicated that changes in the LPC elicited while experiencing the expressers' emotion predicted the subsequent shifts in closeness through alterations in emotional contagion of the anger emotion in the winning context and the happy emotion in the loss context. This study provides empirical evidence regarding the social function of emotional contagion and proves for the first time that the reward context plays a role in it.

How video calls affect mimicry and trust during interactions

Many social species, humans included, mimic emotional expressions, with important consequences for social bonding. Although humans increasingly interact via video calls, little is known about the effect of these online interactions on the mimicry of scratching and yawning, and their linkage with trust. The current study investigated whether mimicry and trust are affected by these new communication media. Using participant-confederate dyads (n = 27), we tested the mimicry of four behaviours across three different conditions: watching a pre-recorded video, online video call, and face-to-face. We measured mimicry of target behaviours frequently observed in emotional situations, yawn and scratch and control behaviours, lip-bite and face-touch. In addition, trust in the confederate was assessed via a trust game. Our study revealed that (i) mimicry and trust did not differ between face-to-face and video calls, but were significantly lower in the pre-recorded condition; and (ii) target behaviours were significantly more mimicked than the control behaviours. This negative relationship can possibly be explained by the negative connotation usually associated with the behaviours included in this study. Overall, this study showed that video calls might provide enough interaction cues for mimicry to occur in our student population and during interactions between strangers. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Attention Towards Pupil Size in Humans and Bonobos (Pan paniscus)

Previous work has established that humans have an attentional bias towards emotional signals, and there is some evidence that this phenomenon is shared with bonobos, our closest relatives. Although many emotional signals are explicit and overt, implicit cues such as pupil size also contain emotional information for observers. Pupil size can impact social judgment and foster trust and social support, and is automatically mimicked, suggesting a communicative role. While an attentional bias towards more obvious emotional expressions has been shown, it is unclear whether this also extends to a more subtle implicit cue, like changes in pupil size. Therefore, the current study investigated whether attention is biased towards pupils of differing sizes in humans and bonobos. A total of 150 human participants (141 female), with a mean age of 19.13 (ranging from 18 to 32 years old), completed an online dot-probe task. Four female bonobos (6 to 17 years old) completed the dot-probe task presented via a touch screen. We used linear mixed multilevel models to examine the effect of pupil size on reaction times. In humans, our analysis showed a small but significant attentional bias towards dilated pupils compared to intermediate-sized pupils and intermediate-sized pupils when compared to small pupils. Our analysis did not show a significant effect in bonobos. These results suggest that the attentional bias towards emotions in humans can be extended to a subtle unconsciously produced signal, namely changes in pupil size. Due to methodological differences between the two experiments, more research is needed before drawing a conclusion regarding bonobos.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-022-00146-1.

Reading Your Emotions in My Physiology? Reliable Emotion Interpretations in Absence of a Robust Physiological Resonance

Affective states are expressed in an individual’s physical appearance, ranging from facial expressions and body postures, to indicators of physiological arousal (e.g., a blush). Confirming the claimed communicative function of these markers, humans are capable of distinguishing between a variety of discrete emotion displays. In an attempt to explain the underlying mechanism, characteristic bodily changes within the observer, including physiological arousal and mimicry, have been suggested to facilitate the interpretation of an expression. The current study aims to create a holistic picture of emotion perception by (1) using three different sources of emotional information (prototypical facial expressions, bodily expressions, and subtle facial cues) and (2) measuring changes in multiple physiological signals (facial electromyography, skin conductance level, skin temperature, and pupil size). While participants clearly discriminated between perceived emotional expressions, there was no overall 1–1 correspondence with their physiological responses. Some specific but robust effects were observed. Angry facial expressions were consistently responded to with a peak in skin conductance level. Furthermore, sad body expressions were associated with a drop in skin temperature. In addition to being the best recognized expression, viewing happy faces elicited congruent facial muscle responses, which supports the potential role of embodied simulation in emotion recognition. Lastly, tears were not only rated as highly emotional intense but also evoked a peak in skin conductance level in the observer. The absence of distinct physiological responses to other expressions could be explained by the lacking functionality of affect sharing in a non-interactive experimental context. Consequentially, emotional alignment in body and mind might especially take place in real social situations, which should be considered in future research.

Trust Measurement in Human-Autonomy Teams: Development of a Conceptual Toolkit

The rise in artificial intelligence capabilities in autonomy-enabled systems and robotics has pushed research to address the unique nature of human-autonomy team collaboration. The goal of these advanced technologies is to enable rapid decision making, enhance situation awareness, promote shared understanding, and improve team dynamics. Simultaneously, use of these technologies is expected to reduce risk to those who collaborate with these systems. Yet, for appropriate human- autonomy teaming to take place, especially as we move beyond dyadic partnerships, proper calibration of team trust is needed to effectively coordinate interactions during high-risk operations. But to meet this end, critical measures of team trust for this new dynamic of human-autonomy teams are needed. This paper seeks to expand on trust measurement principles and the foundation of human-autonomy teaming to propose a “toolkit” of novel methods that support the development, maintenance and calibration of trust in human-autonomy teams operating within uncertain, risky, and dynamic environments.

Can third-party observers detect attraction in others based on subtle nonverbal cues?

In a series of three studies, we examined whether third-party observers can detect attraction in others based on subtle nonverbal cues. We employed video segments of dates collected from a speed-dating experiment, in which daters went on a brief (approx. 4 min) blind-date and indicated whether they would like to go on another date with their brief interaction partner or not. We asked participants to view these stimuli and indicate whether or not each couple member is attracted to their partner. Our results show that participants could not reliably detect attraction, and this ability was not influenced by the age of the observer, video segment location (beginning or middle of the date), video duration, or general emotion recognition capacity. Contrary to previous research findings, our findings suggest that third-party observers cannot reliably detect attraction in others. However, there was one exception: Recognition rose above chance level when the daters were both interested in their partners compared to when they were not interested.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12144-022-02927-0.

It Is Written in the Eyes: Inferences From Pupil Size and Gaze Orientation Shape Interpersonal Liking

Research has shown that pupil size shapes interpersonal impressions: Individuals with dilated pupils tend to be perceived more positively than those with constricted pupils. Untested so far is the role of cognitive processes in shaping the effects of pupil size. Two preregistered studies investigated whether the effect of pupil size was qualified by partner's attention allocation inferred from gaze orientation. In Experiment 1 (N = 50) partners with dilated pupils were more liked when gazing toward the participant, but less liked when gazing toward a disliked other. Experiment 2 (N = 50) unveiled the underlying mechanism of the pupil-gaze interplay. Pupillary changes led to inferences about the feelings held by the partner toward the gazed target: Larger pupils signaled positive feelings. Crucially, target identity moderated the response of the participants (i.e., liking toward the partner). This work shows the importance of considering the interplay of affective and cognitive eye-signals when studying person perception.

Contagious Depression: Automatic Mimicry and the Mirror Neuron System - A Review

Contagious depression is a theory proposing that depression can be induced or triggered by our social environment. This theory is based on emotional contagion, the idea that affective states can be transferred during social interaction, since humans can use emotional contagion to communicate feelings and emotions in conscious and unconscious ways. This review presents behavioral, physiological, and neuroanatomical aspects of two essential contagious depression mechanisms, automatic mimicry and the mirror neuron system.

Building and validation of a set of facial expression images to detect emotions: a transcultural study

The automatic emotion recognition from facial expressions has become an exceptional tool in research involving human subjects and has made it possible to obtain objective measurements of the emotional state of research subjects. Different software and commercial solutions are offered to perform this task. However, the adaptation to cultural context and the recognition of complex expressions and/or emotions are two of the main challenges faced by these solutions. Here, we describe the construction and validation of a set of facial expression images suitable for training a recognition system. Our datasets consist of images of people with no experience in acting who were recorded with a webcam as they performed a computer-assisted task in a room with a light background and overhead illumination. The six basic emotions and mockery were included and a combination of OpenCV, Dlib and Scikit-learn Python libraries were used to develop a support vector machine classifier. The code is available at GitHub and the images will be provided upon request. Since transcultural facial expressions to evaluate complex emotions and open-source solutions were used in this study, we strongly believe that our dataset will be useful in different research contexts.

Emotional Contagion: A Brief Overview and Future Directions

Social interactions can trigger emotional contagion between individuals resulting in behavioral synchrony. Emotional contagion can be a very effective and attractive strategy in communication and advertising, and understanding the mechanisms underlying emotional contagion can help marketers to improve their commercial approaches or develop better ones. The purpose of this study is to review and classify the various methodologies and theoretical approaches on emotional contagion, identify the best practices in this domain, and identify ways of gaging and measuring emotional contagion. The study is based on a mini literature review. We identify different mechanisms and approaches to emotional contagion described in the literature. Emotional contagion can be triggered by facial expressions, indirect human interactions, and/or by observing other people's behavior in direct and indirect interactions. Furthermore, emotional contagion can be triggered physiologically or neurologically by synchronizing with the emotional state of others during human interactions. Regarding the assessment and measurement of emotional contagion, we argue that methods based on neuroscience tools are much more accurate and effective than methods based on traditional research approaches. The study identifies guidelines for research on commercial communication through emotional contagion that can be especially interesting for academia and marketing practitioners. The findings are important for field marketers interested in developing new individualized approaches in their commercial strategies and marketing in general. In addition, the study can become the basis of research that further refines and compares the efficacy of the various techniques and tools involved.

Attentional Bias in Humans Toward Human and Bonobo Expressions of Emotion

Correctly recognizing and efficiently attending to emotional situations are highly valuable skills for social species such as humans and bonobos, humans' closest living relatives. In the current study, we investigated whether humans perceive a range of emotional situations differently when these involved other humans compared to bonobos. A large group of children and adults participated in an emotion perception task and rated scenes showing either bonobos or humans in situations depicting distressed or aggressive behavior, yawning, scratching, grooming, playing, sex scenes or neutral situations. A new group of people performed a dot-probe task to assess attentional biases toward these materials. The main finding is that humans perceive emotional scenes showing people similarly as emotional scenes of bonobos, a result reflecting a shared evolutionary origin of emotional expressions. Other results show that children interpreted bonobos' bared teeth displays as a positive signal. This signal is related to the human smile, but is frequently seen in distressed situations, as was the case in the current experiment. Children may still need to learn to use contextual cues when judging an ambiguous expression as positive or negative. Further, the sex scenes were rated very positively, especially by male participants. Even though they rated these more positively than women, their attention was captured similarly, surpassing all other emotion categories. Finally, humans' attention was captured more by human yawns than by bonobo yawns, which may be related to the highly contagious nature of yawns, especially when shown by close others. The current research adds to earlier work showing morphological, behavioral and genetic parallels between humans and bonobos by showing that their emotional expressions have a common origin too.

When animals cry: The effect of adding tears to animal expressions on human judgment

At a time of growing interest in and awareness about the relationships between humans and animals, it is of relevance to scientifically analyse the intrinsic nature of these interactions. Reactions to emotional tears show our extraordinary capacity for detecting micro-nuances when judging another human’s face. Regarding such behaviour, previous studies carried out in our laboratory have pointed to an adaptive function of emotional tears: i.e. their inhibitory influence on perceived aggressiveness. In the present work we aimed to further explore that hypothesis by extending our investigation from humans to animals, using pictures of five different animal faces (cat, dog, horse, chimpanzee, hamster) to which tears were added digitally. To this end, we conducted an online study of 403 participants recruited from different social networks and academic institutions. We questioned the participants about their perceptions of emotional intensity, aggressiveness and friendliness in the animal faces and analysed the comparisons they made between faces with and without tears. In addition, a latent variable referred to as “passion for animals” was measured using different indicators. By adding the results obtained in each species and breaking them down into different basic emotions, we found that the presence of tears was related to a higher absolute frequency of participants who perceived sadness, which endorsed our previous results obtained using images of humans. Regarding aggressiveness, the presence of tears favoured the perception of less aggressiveness. A structural equation model was also conducted to explore the relations among all the measured variables. The model confirmed that the presence of tears in the animal faces had a significant influence on the perception of higher emotional intensity and friendliness, and of lower aggressiveness.

Human and Human-Interfaced AI Interactions: Modulation of Human Male Autonomic Nervous System via Pupil Mimicry

Pupillary alterations in virtual humans induce neurophysiological responses within an observer. Technological advances have enabled rapid developments in artificial intelligence (AI), from verbal systems, to visual AI interfaces with the ability to express, and respond to emotional states of a user. Visual AI interfaces are able to change their physical parameters, such as pupil diameter. Pupillary changes can alter heart rate, however, effects on heart rate variability (HRV) are unknown. HRV, is an autonomic, non-conscious parameter which monitors sympathetic and parasympathetic nervous system (PNS) activity. N = 34 male participants aged between 19–33 were subjected to a number of conditions such as pupil dilation, constriction and blushing. The present research is the first to investigate the effects of virtual human interactions on human HRV. Outcomes of this study were obtained using eye tracking and HRV measurements. Pupil dilation relative to constriction presented in the female virtual partner induced a significant right pupillary diameter increase (p = 0.041) in human observers. Additionally, female virtual partner pupil constriction relative to dilation induced a significant increase in participants’ PNS HRV response (p = 0.036). These findings indicate the ability of a female virtual interaction partner to modulate parasympathetic autonomic functioning in young healthy male humans. This allows first insights into the effects of interacting with virtual AI interaction partners, on human autonomic functioning, and may aid development of future virtual humans, and their implementation into relevant clinical settings.

Emotion recognition in nonhuman primates: How experimental research can contribute to a better understanding of underlying mechanisms

Recognising conspecifics' emotional expressions is important for nonhuman primates to navigate their physical and social environment. We address two possible mechanisms underlying emotion recognition: emotional contagion, the automatic matching of the observer's emotions to the emotional state of the observed individual, and cognitive empathy, the ability to understand the meaning and cause of emotional expressions while maintaining a distinction between own and others' emotions. We review experimental research in nonhuman primates to gain insight into the evolution of emotion recognition. Importantly, we focus on how emotional contagion and cognitive empathy can be studied experimentally. Evidence for aspects of cognitive empathy in different nonhuman primate lineages suggests that a wider range of primates than commonly assumed can infer emotional meaning from emotional expressions. Possibly, analogous rather than homologous evolution underlies emotion recognition. However, conclusions regarding its exact evolutionary course require more research in different modalities and species.

Feeling Stressed and Ugly? Leave the City and Visit Nature! An Experiment on Self-and Other-Perceived Stress and Attractiveness Levels

Natural environments, compared to urban environments, usually lead to reduced stress and positive body appreciation. We assumed that walks through nature and urban environments affect self- and other-perceived stress and attractiveness levels. Therefore, we collected questionnaire data and took photographs of male participants' faces before and after they took walks. In a second step, female participants rated the photographs. As expected, participants felt more restored and attractive, and less stressed after they walked in nature compared to an urban environment. A significant interaction of environment (nature, urban) and time (pre, post) indicated that the men were rated by the women as being more stressed after the urban walk. Other-rated attractiveness levels, however, were similar for both walks and time points. In sum, we showed that the rather stressful experience of a short-term urban walk mirrors in the face of men and is detectable by women.

Physiological synchrony is associated with cooperative success in real-life interactions

Cooperation is pivotal for society to flourish. To foster cooperation, humans express and read intentions via explicit signals and subtle reflections of arousal visible in the face. Evidence is accumulating that humans synchronize these nonverbal expressions and the physiological mechanisms underlying them, potentially influencing cooperation. The current study is designed to verify this putative linkage between synchrony and cooperation. To that end, 152 participants played the Prisoner's Dilemma game in a dyadic interaction setting, sometimes facing each other and sometimes not. Results showed that synchrony in both heart rate and skin conductance level emerged during face-to-face contact. However, only synchrony in skin conductance levels predicted cooperative success of dyads. Crucially, this positive linkage was strengthened when participants could see each other. These findings show the strong relationship between our bodily responses and social behavior, and emphasize the importance of studying social processes between rather than within individuals in real-life interactions.

Mirror replication of sexual facial expressions increases the success of sexual contacts in bonobos

Rapid Facial Mimicry (RFM), one of the possible predictors of emotional contagion, is defined as the rapid, involuntary and automatic replication of a facial expression. Up to now, RFM has been demonstrated in nonhuman animals exclusively during play. Since in bonobos, as in humans, socio-sexuality is a powerful tool for assessing/strengthening inter-individual relationships, we investigated RFM in this domain. Bonobos displayed silent bared-teeth (sbt, the most common facial expression during sexual contacts) more frequently after the detection of an sbt emitted by the trigger than in the no-detection condition. This is the first demonstration of the presence of RFM during sex. The occurrence of RFM was positively affected by the sex of the partners with female homo-sexual contacts being punctuated by a higher presence of RFM. At an immediate level, RFM increased the duration of homo- and hetero-sexual contacts. This finding suggests that RFM can increase individuals’ potential fitness benefits. By prolonging their sexual contacts, females can strengthen their social relationships thus increasing the probability to obtain priority over resources (RFM indirect fitness benefits). Via longer copulations, males can increase the probability to make females pregnant (RFM direct fitness benefits). In conclusion, in bonobos the access to the partner’s face during sexual contacts (face-to-face, proximate factor) and the role of socio-sexuality in increasing the individual direct and indirect fitness (ultimate factor) could have favoured the evolution of specific sexual facial expressions and their rapid mirror replication. Our findings on bonobos expand the role of RFM well beyond the animal play domain thus opening new scenarios for future comparative studies exploring the evolution of socio-sexuality in humans.

Clinician-Patient Movement Synchrony Mediates Social Group Effects on Interpersonal Trust and Perceived Pain

Pain is an unfortunate consequence of many medical procedures, which in some patients becomes chronic and debilitating. Among the factors affecting medical pain, clinician-patient (C-P) similarity and nonverbal communication are particularly important for pain diagnosis and treatment. Participants (N = 66) were randomly assigned to clinician and patient roles and were grouped into C-P dyads. Clinicians administered painful stimuli to patients as an analogue of a painful medical procedure. We manipulated the perceived C-P similarity of each dyad using groups ostensibly based on shared beliefs and values, and each patient was tested twice: Once with a same group clinician (concordant, CC) and once with a clinician from the other group (discordant, DC). Movement synchrony was calculated as a marker of nonverbal communication. We tested whether movement synchrony mediated the effects of group concordance on patients' pain and trust in the clinician. Movement synchrony was higher in CC than DC dyads. Higher movement synchrony predicted reduced pain and increased trust in the clinician. Movement synchrony also formally mediated the group concordance effects on pain and trust. These findings increase our understanding of the role of nonverbal C-P communication on pain and related outcomes. Interpersonal synchrony may be associated with better pain outcomes, independent of the specific treatment provided. PERSPECTIVE: This article demonstrates that movement synchrony in C-P interactions is an unobtrusive measure related to their relationship quality, trust toward the clinician, and pain. These findings suggest that interpersonal synchrony may be associated with better patient outcomes, independent of the specific treatment provided.

How Our Gaze Reacts to Another Person's Tears? Experimental Insights Into Eye Tracking Technology

Crying is an ubiquitous human behavior through which an emotion is expressed on the face together with visible tears and constitutes a slippery riddle for researchers. To provide an answer to the question "How our gaze reacts to another person's tears?," we made use of eye tracking technology to study a series of visual stimuli. By presenting an illustrative example through an experimental setting specifically designed to study the "tearing effect," the present work aims to offer methodological insight on how to use eye-tracking technology to study non-verbal cues. A sample of 30 healthy young women with normal visual acuity performed a within-subjects task in which they evaluated images of real faces with and without tears while their eye movements were tracked. Tears were found to be a magnet for visual attention in the task of facial attribution, facilitating a greater perception of emotional intensity. Moreover, the inspection pattern changed qualitatively and quantitatively, with our participants becoming fully focused on the tears when they were visible. The mere presence of a single tear running down a cheek was associated with an increased emotional inference and greater perception of sincerity. Using normalized and validated tools (Reading the Eyes in the Mind Test and the SALAMANCA screening test for personality disorders), we measured the influence of certain characteristics of the participants on their performance of the experimental task. On the one hand, a higher level of cognitive empathy helped to classify tearful faces with higher emotional intensity and tearless faces with less emotional intensity. On the other hand, we observed that less sincerity was attributed to the tearful faces as the SALAMANCA test scores rose in clusters A (strange and extravagant) and B (immature and emotionally unstable) of our sample. The present findings highlight the advantages of using eye tracking technology to study non-verbal cues and draw attention to methodological issues that should be taken into account. Further exploration of the relationship between empathy and tear perception could be a fruitful avenue of future research using eye tracking.

A sensorimotor control framework for understanding emotional communication and regulation

Our research team was asked to consider the relationship of the neuroscience of sensorimotor control to the language of emotions and feelings. Actions are the principal means for the communication of emotions and feelings in both humans and other animals, and the allostatic mechanisms controlling action also apply to the regulation of emotional states by the self and others. We consider how motor control of hierarchically organised, feedback-based, goal-directed action has evolved in humans, within a context of consciousness, appraisal and cultural learning, to serve emotions and feelings. In our linguistic analysis, we found that many emotion and feelings words could be assigned to stages in the sensorimotor learning process, but the assignment was often arbitrary. The embodied nature of emotional communication means that action words are frequently used, but that the meanings or senses of the word depend on its contextual use, just as the relationship of an action to an emotion is also contextually dependent.

Pupil mimicry in infants and parents

Changes in pupil size can reflect social interest or affect, and tend to get mimicked by observers during eye contact. Pupil mimicry has recently been observed in young infants, whereas it is unknown whether the extent and the speed of infants' pupil mimicry response are identical to that of adults. Moreover, the question of whether pupil mimicry in infants is modulated by the race of the observed other remains to be explored. In two studies, pupil mimicry was investigated in infants and their parents. In the first study, 6-, 12- and 18-month-olds (n = 194) and their parents (n = 192) observed eyes with dynamically dilating, constricting, or static pupils. Infants mimicked the pupil sizes of the observed eyes like their parents, but responded slower. Study 2 replicated these findings in a new sample of infants (n = 55, 12-month-olds) and parents (n = 64), and further showed that the pupil mimicry response was not significantly modulated by the race of the observed partner neither in infants nor in parents. We conclude that pupil mimicry is an ancient bonding mechanism that helps to connect people.

Atypical Social Attention and Emotional Face Processing in Autism Spectrum Disorder: Insights From Face Scanning and Pupillometry

Social attention deficits are a hallmark characteristic within autism spectrum disorder (ASD) and have been hypothesized to have cascading effects on emotion recognition. Eye-tracking methodology has emerged as a potentially reliable, feasible, and sensitive biomarker for examining core phenotypic features of ASD; however, these findings are mixed with regards to measuring treatment change in clinical trials. The present study aimed to assess the utility of an eye-tracking paradigm to discriminate between clinical groups in social attention and emotion recognition through face scanning and pupillometry. The present study also assessed the reliability of this paradigm within the ASD sample to further our understanding of the utility of eye-tracking for future clinical trials. Participants included 42 individuals with ASD, 29 developmental disability (DD) controls, and 62 typically developing (TD) controls between 3 and 25 years of age. An emotional faces eye-tracking paradigm was administered to all participants, with the ASD group completing the paradigm a second time approximately 2 months later. Participants' average proportion of looking and number of fixations to specific areas of interest (AOI) were examined along with changes in pupil reactivity while viewing different emotional faces. Results suggest atypical face-scanning through a reduced proportion of looking and the number of fixations toward the eyes in the ASD group regardless of the emotion that was presented. Further, pupillometry measures were able to detect increases in pupil dilation to happy faces in the ASD group. Lastly, test-retest reliability coefficients varied between the poor and excellent range based on the mechanism assessed, with the proportion of looking demonstrating the highest reliability coefficients. These findings build on the promise of eye-tracking as a feasible and reliable biomarker for identifying social attention and emotion recognition deficits in ASD. Detecting differences in emotion recognition explicitly through facial scanning was not as clear. Specific mechanisms within the eye-tracking paradigm may be viable options for assessing treatment-specific outcomes.

Preprocessing pupil size data: Guidelines and code

Pupillometry has been one of the most widely used response systems in psychophysiology. Changes in pupil size can reflect diverse cognitive and emotional states, ranging from arousal, interest and effort to social decisions, but they are also widely used in clinical practice to assess patients’ brain functioning. As a result, research involving pupil size measurements has been reported in practically all psychology, psychiatry, and psychophysiological research journals, and now it has found its way into the primatology literature as well as into more practical applications, such as using pupil size as a measure of fatigue or a safety index during driving. The different systems used for recording pupil size are almost as variable as its applications, and all yield, as with many measurement techniques, a substantial amount of noise in addition to the real pupillometry data. Before analyzing pupil size, it is therefore of crucial importance first to detect this noise and deal with it appropriately, even prior to (if need be) resampling and baseline-correcting the data. In this article we first provide a short review of the literature on pupil size measurements, then we highlight the most important sources of noise and show how these can be detected. Finally, we provide step-by-step guidelines that will help those interested in pupil size to preprocess their data correctly. These guidelines are accompanied by an open source MATLAB script (available at https://github.com/ElioS-S/pupil-size). Given that pupil diameter is easily measured by standard eyetracking technologies and can provide fundamental insights into cognitive and emotional processes, it is hoped that this article will further motivate scholars from different disciplines to study pupil size.

The Interplay Between Face-to-Face Contact and Feedback on Cooperation During Real-Life Interactions

Cooperation forms the basis of our society and becomes increasingly essential during times of globalization. However, despite technological developments people still prefer to meet face-to-face, which has been shown to foster cooperation. However, what is still unclear is how this beneficial effect depends on what people know about their interaction partner. To examine this question, 58 dyads played an iterated Prisoner’s Dilemma game, sometimes facing each other, sometimes without face contact. Additionally, explicit feedback regarding their decisions was manipulated between dyads. The results revealed that participants were more cooperative when they saw each other compared to when they could not, and when receiving reliable compared to unreliable or no feedback. Contradicting our hypothesis that participants would rely more on nonverbal communication in the absence of explicit information, we observed that the two sources of information operated independently on cooperative behavior. Interestingly, although individuals mostly relied on explicit information if available, participants still cooperated more after their partner defected with face-to-face contact compared to no face-to-face contact. The results of our study have implications for real-life interactions, suggesting that face-to-face contact has beneficial effects on prosocial behavior even if people cannot verify whether their selfless acts are being reciprocated.

Is there a prediction network? Meta-analytic evidence for a cortical-subcortical network likely subserving prediction

Predictive coding is an increasingly influential and ambitious concept in neuroscience viewing the brain as a 'hypothesis testing machine' that constantly strives to minimize prediction error, the gap between its predictions and the actual sensory input. Despite the invaluable contribution of this framework to the formulation of brain function, its neuroanatomical foundations have not been fully defined. To address this gap, we conducted activation likelihood estimation (ALE) meta-analysis of 39 neuroimaging studies of three functional domains (action perception, language and music) inherently involving prediction. The ALE analysis revealed a widely distributed brain network encompassing regions within the inferior and middle frontal gyri, anterior insula, premotor cortex, pre-supplementary motor area, temporoparietal junction, striatum, thalamus/subthalamus and the cerebellum. This network is proposed to subserve domain-general prediction and its relevance to motor control, attention, implicit learning and social cognition is discussed in light of the predictive coding scheme. Better understanding of the presented network may help advance treatments of neuropsychiatric conditions related to aberrant prediction processing and promote cognitive enhancement in healthy individuals.

Pupils say more than a thousand words: Pupil size reflects how observed actions are interpreted

Humans attend to others' facial expressions and body language to better understand their emotions and predict goals and intentions. The eyes and its pupils reveal important social information. Because pupil size is beyond voluntary control yet reflective of a range of cognitive and affective processes, pupils in principal have the potential to convey whether others' actions are interpreted correctly or not. Here, we measured pupil size while participants observed video-clips showing reach-to-grasp arm movements. Expressors in the video-clips were playing a board game and moved a dowel to a new position. Participants' task was to decide whether the dowel was repositioned with the intention to be followed up by another move of the same expressor (personal intention) or whether the arm movement carried the implicit message that expressor's turn was over (social intention). Replicating earlier findings, results showed that participants recognized expressors' intentions on the basis of their arm kinematics. Results further showed that participants' pupil size was larger when observing actions reflecting personal compared to social intentions. Most interestingly, before participants indicated how they interpreted the observed actions by choosing to press one of two keys (corresponding to the personal or social intention), their pupils within a split second, had already given away how they interpreted the expressor's movement. In sum, this study underscores the importance of nonverbal behavior in helping social messages get across quickly. Revealing how actions are interpreted, pupils may provide additional feedback for effective social interactions.

Spontaneous blink rate as an index of attention and emotion during film clips viewing

Spontaneous blinking is a non-invasive indicator known to reflect dopaminergic influence over frontal cortex and attention allocation in perceptual tasks. 38 participants watched eighteen short film clips (2 min), designed to elicit specific affective states, and arranged in six different emotional categories, while their eye movements were recorded from the vertical electroculogram. The largest blink rate inhibition, reflecting greater attention allocation to the movie, was observed during the presentation of Erotic clips, excerpts on wilderness depicting beautiful landscapes (Scenery), as well as clips showing crying characters (Compassion). Instead, the minimum blink rate inhibition was found for Fear clips, which induced a defensive response with stimulus rejection. Blink rate across time evidenced how Compassion clips elicited early inhibition while Sadness clips induced a slower, later inhibition. Correlation analyses also revealed a negative correlation (r < -0.40) between total blink rate recorded during Erotic and Compassion clips and self-reported interest. Overall, the main variable explaining blink rate was emotional Valence. Results suggest that blink modulation is related with the motivational relevance and biological significance of the stimuli, tracking their differential recruitment of attentional resources. Furthermore, they provide a solid background for studying the emotion-attention patterns and their deficits also in clinical samples (e.g., neurological and psychiatric patients) using spontaneous blinking as a not-interfering psychophysiological measure.

Normative surface skin temperature changes due to blood redistribution: A prospective study

The continuing development and manufacture of infrared devices, together with improvements in thermal body mapping techniques have simplified surface skin thermography which is being used more extensively than ever before. Normative thermography data, however, remains incomplete. A normative blood redistribution range of skin temperatures was established for use as a reference for laboratory infrared thermography (IT), thermal body mapping, and mass fever screenings. 500 healthy volunteers participated in this prospective study. To determine the maximum range of the skin temperature changes due to the posture-related physiological blood redistribution, the volunteers were asked to keep one extremity up and another extremity down whilst lying, sitting, and standing. We obtained 6000 hand and 400 foot temperature readings. The normal temperature was 29.1 ± 0.6 °C for the middle fingers and 27.8 ± 0.7 °C for the toes. The physiological temperature change during body position changes ranged from 4 to 6 °C (fingers: 27-31 °C; toes: 26-32 °C). At normal room temperature, the surface skin temperature may vary within this range due to blood redistribution. These changes reflect the individual variability of vasomotor activity. This physiological range of temperatures should be taken into account during IT and other thermography-involved investigations.

Masked ambiguity - Emotion identification in schizophrenia and major depressive disorder

Both patients with schizophrenia and with a major depressive disorder (MDD) display deficits in identifying facial expressions of emotion during acute phases of their illness. However, specific deficit patterns have not yet been reliably demonstrated. Tasks that employ emotionally ambiguous stimuli have recently shown distinct deficit patterns in patients with schizophrenia compared to other mental disorders as well as healthy controls. We here investigate whether a task which uses an ambiguous Japanese (Noh) mask and a corresponding human stimulus generates distinctive emotion attribution patterns in thirty-two Caucasian patients with schizophrenia, matched MDD patients and healthy controls. Results show that patients with schizophrenia displayed reaction time disadvantages compared to healthy controls while identifying sadness and anger. MDD patients were more likely to label stimuli with basic compared to subtle emotional expressions. Moreover, they showed more difficulties assigning emotions to the human stimulus than to the Noh mask. IQ, age and cognitive functioning did not modulate these results. Because overall group differences were not observed, this task is not suitable for diagnosing patients. However, the subtle differences that did emerge might give therapists handles that can be used in therapy.

The influence of facial expressions on attack recognition and response behaviour in karate kumite

Facial expressions play a crucial role in human daily life interactions, so it is possible that they contain relevant anticipatory cues. Therefore, we conducted two studies in karate kumite to analyse the influence of facial expressions on the attack recognition (study A) and on the response behaviour (study B). In study A, 10 karate athletes (age 14-26 years, national and international level) had to recognize temporally occluded karate attacks in 2 conditions (normal and blurred face of the attacker) and identify the upcoming attack (verbal reports). In study B, a natural fight was initiated with 12 karate athletes (age 15-32 years, national and international level) in 3 conditions: an attacker with a normal face, wearing a ski mask, and wearing a ski mask together with sunglasses. The measured parameters response quality and time for response were calculated applying the McNemar test (study A) and Friedman test (study B). Results of study A showed a significant improvement in the attack recognition in the blurred face condition compared to the normal face condition. In study B a significant improvement in time for response was observed in the ski mask and ski mask plus sunglasses conditions (r < 0.1). No significant difference for response quality was found in any of the three conditions. The results of our work indicate that the disguise of the attacker's face leads to a better attack recognition and response behaviour. We conclude that anticipatory cues in karate kumite context mainly lie in the attacker's movements and not in the attacker's face.

Jumping for Joy: The Importance of the Body and of Dynamics in the Expression and Recognition of Positive Emotions

The majority of research on emotion expression has focused on static facial prototypes of a few selected, mostly negative emotions. Implicitly, most researchers seem to have considered all positive emotions as sharing one common signal (namely, the smile), and consequently as being largely indistinguishable from each other in terms of expression. Recently, a new wave of studies has started to challenge the traditional assumption by considering the role of multiple modalities and the dynamics in the expression and recognition of positive emotions. Based on these recent studies, we suggest that positive emotions are better expressed and correctly perceived when (a) they are communicated simultaneously through the face and body and (b) perceivers have access to dynamic stimuli. Notably, we argue that this improvement is comparatively more important for positive emotions than for negative emotions. Our view is that the misperception of positive emotions has fewer immediate and potentially life-threatening consequences than the misperception of negative emotions; therefore, from an evolutionary perspective, there was only limited benefit in the development of clear, quick signals that allow observers to draw fine distinctions between them. Consequently, we suggest that the successful communication of positive emotions requires a stronger signal than that of negative emotions, and that this signal is provided by the use of the body and the way those movements unfold. We hope our contribution to this growing field provides a new direction and a theoretical grounding for the many lines of empirical research on the expression and recognition of positive emotions.

Looking into your eyes: observed pupil size influences approach-avoidance responses

The eyes reveal important social messages, such as emotions and whether a person is aroused and interested or bored and fatigued. A growing body of research has also shown that individuals with large pupils are generally evaluated positively by observers, while those with small pupils are perceived negatively. Here, we examined whether observed pupil size influences approach-avoidance tendencies. Participants performed an Approach-Avoidance Task using faces with large and small pupil sizes. Results showed that pupil size influences the accuracy of arm movements. Specifically, individuals were less prone to approach a face with small pupils than a face with large pupils. Conversely, participants were less prone to avoid a face with large pupils than a face with small pupils. Collectively, these findings suggest that perceivers attend to a facial cue - pupil size - when interacting with others.

Pupil mimicry and trust - Implication for depression

Individuals suffering from depression often have difficulty trusting others. Previous research has shown a relationship between trust formation and pupil mimicry - the synchronization of pupil sizes between individuals. The current study therefore examined whether pupil mimicry is weaker in depressed individuals and an underlying factor of their low levels of trust. Forty-two patients with major depressive disorder (MDD) and 40 healthy control subjects played trust games with virtual partners. Images of these partners' eye regions were presented to participants before they had to make a monetary investment decision. Partners' pupils either dilated, constricted, or remained static over the course of 4-s interactions. During the task, participants' pupil sizes were recorded with eye-tracking equipment to assess mimicry. The results confirm that patients with MDD were somewhat less trusting than controls and used another's pupillary cues differently when deciding to trust. Specifically, whereas healthy controls trusted partners with dilating pupils more than partners with constricting pupils, patients with MDD particularly trusted partners whose pupils changed in size less, regardless of whether partners' pupils were dilating or constricting. This difference in investment behavior was unrelated to differences in pupil mimicry, which was equally apparent in both groups and fostered trust to the same extent. Whereas lower levels of trust observed in patients with MDD could not be explained by differences in pupil mimicry, our data show that pupil dilation mimicry might help people to trust. These findings provide further evidence for the important role of pupil size and pupil mimicry in interpersonal trust formation and shed light on the pathophysiology of clinically low trust in patients with MDD.

Pupil-mimicry conditions trust in partners: moderation by oxytocin and group membership

Across species, oxytocin, an evolutionarily ancient neuropeptide, facilitates social communication by attuning individuals to conspecifics' social signals, fostering trust and bonding. The eyes have an important signalling function; and humans use their salient and communicative eyes to intentionally and unintentionally send social signals to others, by contracting the muscles around their eyes and pupils. In our earlier research, we observed that interaction partners with dilating pupils are trusted more than partners with constricting pupils. But over and beyond this effect, we found that the pupil sizes of partners synchronize and that when pupils synchronously dilate, trust is further boosted. Critically, this linkage between mimicry and trust was bound to interactions between ingroup members. The current study investigates whether these findings are modulated by oxytocin and sex of participant and partner. Using incentivized trust games with partners from ingroup and outgroup whose pupils dilated, remained static or constricted, this study replicates our earlier findings. It further reveals that (i) male participants withhold trust from partners with constricting pupils and extend trust to partners with dilating pupils, especially when given oxytocin rather than placebo; (ii) female participants trust partners with dilating pupils most, but this effect is blunted under oxytocin; (iii) under oxytocin rather than placebo, pupil dilation mimicry is weaker and pupil constriction mimicry stronger; and (iv) the link between pupil constriction mimicry and distrust observed under placebo disappears under oxytocin. We suggest that pupil-contingent trust is parochial and evolved in social species in and because of group life.

Motor signatures of emotional reactivity in frontotemporal dementia

Automatic motor mimicry is essential to the normal processing of perceived emotion, and disrupted automatic imitation might underpin socio-emotional deficits in neurodegenerative diseases, particularly the frontotemporal dementias. However, the pathophysiology of emotional reactivity in these diseases has not been elucidated. We studied facial electromyographic responses during emotion identification on viewing videos of dynamic facial expressions in 37 patients representing canonical frontotemporal dementia syndromes versus 21 healthy older individuals. Neuroanatomical associations of emotional expression identification accuracy and facial muscle reactivity were assessed using voxel-based morphometry. Controls showed characteristic profiles of automatic imitation, and this response predicted correct emotion identification. Automatic imitation was reduced in the behavioural and right temporal variant groups, while the normal coupling between imitation and correct identification was lost in the right temporal and semantic variant groups. Grey matter correlates of emotion identification and imitation were delineated within a distributed network including primary visual and motor, prefrontal, insular, anterior temporal and temporo-occipital junctional areas, with common involvement of supplementary motor cortex across syndromes. Impaired emotional mimesis may be a core mechanism of disordered emotional signal understanding and reactivity in frontotemporal dementia, with implications for the development of novel physiological biomarkers of socio-emotional dysfunction in these diseases.