Transmission of facial expressions of emotion co-evolved with their efficient decoding in the brain: behavioral and brain evidence

Relevance of pre-stimulus oscillatory activity for the perceived valence of emotional facial expressions

The interpretation of emotional facial expressions is crucial in everyday social interactions, and rapid processing of these expressions is necessary. Although extensive research has shed light on the mechanisms involved in facial expression processing, there is limited research on the potential role of the state of neural activity that directly precedes the occurrence of a face. Here, we investigated the potential modulatory role of pre-stimulus oscillatory activity in emotional facial expression processing. We tested emotional facial processing in two experiments, one utilizing artificial and the other natural facial expressions. The participants had to evaluate the emotional valence of the presented ambiguous facial expressions. In a univariate analysis, differences in the oscillation activity of the later rated valence of the faces were observed in both experiments, and these differences were observed even before the presentation of the facial expressions. Importantly, two different multivariate approaches directly supported the relevance of pre-stimulus oscillatory activity by exclusively using pre-stimulus oscillatory data to predict the perceived valence of the latter rated facial expression across the two experiments within as well as across subjects. The behavioral data shows the often observed negativity bias, i.e. ambiguous faces resulted in the tendency to rate them as negative. This negativity bias was related to neural activity modulations in the pre-stimulus period and also within post-stimulus processing related activity. These findings underscore the significance of pre-stimulus oscillatory activity in facial expression processing, indicating a functional role of ongoing neural states that affects the processing of facial expressions and constitute a basis for the well described negativity bias.

Subjective Affective Responses to Natural Scenes Require Understanding, Not Spatial Frequency Bands

It is debated whether emotional processing and response depend on semantic identification or are preferentially tied to specific information in natural scenes, such as global features or local details. The present study aimed to further examine the relationship between scene understanding and affective response while manipulating visual content. To this end, we presented affective and neutral natural scenes which were progressively band-filtered to contain global features (low spatial frequencies) or local details (high spatial frequencies) and assessed both affective response and scene understanding. We observed that, if scene content was correctly reported, subjective ratings of arousal and valence were modulated by the affective content of the scene, and this modulation was similar across spatial frequency bands. On the other hand, no affective modulation of subjective ratings was observed if picture content was not correctly reported. The present results indicate that subjective affective response requires content understanding, and it is not tied to a specific spatial frequency range.

Cultural facial expressions dynamically convey emotion category and intensity information

Communicating emotional intensity plays a vital ecological role because it provides valuable information about the nature and likelihood of the sender's behavior.1,2,3 For example, attack often follows signals of intense aggression if receivers fail to retreat.4,5 Humans regularly use facial expressions to communicate such information.6,7,8,9,10,11 Yet how this complex signaling task is achieved remains unknown. We addressed this question using a perception-based, data-driven method to mathematically model the specific facial movements that receivers use to classify the six basic emotions-"happy," "surprise," "fear," "disgust," "anger," and "sad"-and judge their intensity in two distinct cultures (East Asian, Western European; total n = 120). In both cultures, receivers expected facial expressions to dynamically represent emotion category and intensity information over time, using a multi-component compositional signaling structure. Specifically, emotion intensifiers peaked earlier or later than emotion classifiers and represented intensity using amplitude variations. Emotion intensifiers are also more similar across emotions than classifiers are, suggesting a latent broad-plus-specific signaling structure. Cross-cultural analysis further revealed similarities and differences in expectations that could impact cross-cultural communication. Specifically, East Asian and Western European receivers have similar expectations about which facial movements represent high intensity for threat-related emotions, such as "anger," "disgust," and "fear," but differ on those that represent low threat emotions, such as happiness and sadness. Together, our results provide new insights into the intricate processes by which facial expressions can achieve complex dynamic signaling tasks by revealing the rich information embedded in facial expressions.

Recognizing facial expressions of emotion amid noise: A dynamic advantage

Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.

Detection of emotional faces: The role of spatial frequencies and local features

Models of emotion processing suggest that threat-related stimuli such as fearful faces can be detected based on the rapid extraction of low spatial frequencies. However, this remains debated as other models argue that the decoding of facial expressions occurs with a more flexible use of spatial frequencies. The purpose of this study was to clarify the role of spatial frequencies and differences in luminance contrast between spatial frequencies, on the detection of facial emotions. We used a saccadic choice task in which emotional-neutral face pairs were presented and participants were asked to make a saccade toward the neutral or the emotional (happy or fearful) face. Faces were displayed either in low, high, or broad spatial frequencies. Results showed that participants were better to saccade toward the emotional face. They were also better for high or broad than low spatial frequencies, and the accuracy was higher with a happy target. An analysis of the eye and mouth saliency ofour stimuli revealed that the mouth saliency of the target correlates with participants' performance. Overall, this study underlines the importance of local more than global information, and of the saliency of the mouth region in the detection of emotional and neutral faces.

Potentiated early neural responses to fearful faces are not driven by specific face parts

Prioritized processing of fearful compared to neutral faces is reflected in increased amplitudes of components of the event-related potential (ERP). It is unknown whether specific face parts drive these modulations. Here, we investigated the contributions of face parts on ERPs to task-irrelevant fearful and neutral faces using an ERP-dependent facial decoding technique and a large sample of participants (N = 83). Classical ERP analyses showed typical and robust increases of N170 and EPN amplitudes by fearful relative to neutral faces. Facial decoding further showed that the absolute amplitude of these components, as well as the P1, was driven by the low-frequency contrast of specific face parts. However, the difference between fearful and neutral faces was not driven by any specific face part, as supported by Bayesian statistics. Furthermore, there were no correlations between trait anxiety and main effects or interactions. These results suggest that increased N170 and EPN amplitudes to task-irrelevant fearful compared to neutral faces are not driven by specific facial regions but represent a holistic face processing effect.

Face mediated human–robot interaction for remote medical examination

Realtime visual feedback from consequences of actions is useful for future safety-critical human–robot interaction applications such as remote physical examination of patients. Given multiple formats to present visual feedback, using face as feedback for mediating human–robot interaction in remote examination remains understudied. Here we describe a face mediated human–robot interaction approach for remote palpation. It builds upon a robodoctor–robopatient platform where user can palpate on the robopatient to remotely control the robodoctor to diagnose a patient. A tactile sensor array mounted on the end effector of the robodoctor measures the haptic response of the patient under diagnosis and transfers it to the robopatient to render pain facial expressions in response to palpation forces. We compare this approach against a direct presentation of tactile sensor data in a visual tactile map. As feedback, the former has the advantage of recruiting advanced human capabilities to decode expressions on a human face whereas the later has the advantage of being able to present details such as intensity and spatial information of palpation. In a user study, we compare these two approaches in a teleoperated palpation task to find the hard nodule embedded in the remote abdominal phantom. We show that the face mediated human–robot interaction approach leads to statistically significant improvements in localizing the hard nodule without compromising the nodule position estimation time. We highlight the inherent power of facial expressions as communicative signals to enhance the utility and effectiveness of human–robot interaction in remote medical examinations.

Features and Extra-Striate Body Area Representations of Diagnostic Body Parts in Anger and Fear Perception

Social species perceive emotion via extracting diagnostic features of body movements. Although extensive studies have contributed to knowledge on how the entire body is used as context for decoding bodily expression, we know little about whether specific body parts (e.g., arms and legs) transmit enough information for body understanding. In this study, we performed behavioral experiments using the Bubbles paradigm on static body images to directly explore diagnostic body parts for categorizing angry, fearful and neutral expressions. Results showed that subjects recognized emotional bodies through diagnostic features from the torso with arms. We then conducted a follow-up functional magnetic resonance imaging (fMRI) experiment on body part images to examine whether diagnostic parts modulated body-related brain activity and corresponding neural representations. We found greater activations of the extra-striate body area (EBA) in response to both anger and fear than neutral for the torso and arms. Representational similarity analysis showed that neural patterns of the EBA distinguished different bodily expressions. Furthermore, the torso with arms and whole body had higher similarities in EBA representations relative to the legs and whole body, and to the head and whole body. Taken together, these results indicate that diagnostic body parts (i.e., torso with arms) can communicate bodily expression in a detectable manner.

Human face and gaze perception is highly context specific and involves bottom-up and top-down neural processing

This review summarizes human perception and processing of face and gaze signals. Face and gaze signals are important means of non-verbal social communication. The review highlights that: (1) some evidence is available suggesting that the perception and processing of facial information starts in the prenatal period; (2) the perception and processing of face identity, expression and gaze direction is highly context specific, the effect of race and culture being a case in point. Culture affects by means of experiential shaping and social categorization the way in which information on face and gaze is collected and perceived; (3) face and gaze processing occurs in the so-called 'social brain'. Accumulating evidence suggests that the processing of facial identity, facial emotional expression and gaze involves two parallel and interacting pathways: a fast and crude subcortical route and a slower cortical pathway. The flow of information is bi-directional and includes bottom-up and top-down processing. The cortical networks particularly include the fusiform gyrus, superior temporal sulcus (STS), intraparietal sulcus, temporoparietal junction and medial prefrontal cortex.

Emotional face processing correlates with depression/anxiety symptoms but not wellbeing in non-clinical adults: An event-related potential study

Whilst alterations in emotional face processing, as indicated by event-related potentials (ERPs), are associated with depression and anxiety symptoms in clinical and non-clinical samples, it has remained unclear whether they are related to mental wellbeing. The current study aimed to address this question in a non-clinical sample. The analysis included 402 adult twins from the TWIN-E study. The COMPAS-W and the Depression Anxiety Stress Scale (DASS-42) were used to measure mental wellbeing and depression/anxiety symptoms, respectively. Participants viewed facial expressions under Unmasked (conscious) and Masked (subliminal) conditions while ERPs were recorded. The associations of emotion processing with mental wellbeing and depression/anxiety symptoms were assessed using multivariate linear mixed models. There was a strong association between depression/anxiety symptoms and the N170 amplitude difference for the Fear - Happy contrast in the Masked condition after controlling for wellbeing scores (B = 0.34, p < .001). Specifically, higher depression/anxiety symptoms were associated with a lack of differentiation between fearful and happy faces. No associations were found between emotional face processing and mental wellbeing scores. These results indicate that even within a non-clinical sample, alterations in emotional ERPs, namely the N170, reflect differences in depression/anxiety symptoms rather than differences in wellbeing. Furthermore, this effect was limited to automatic processing, rather than conscious processing of emotional stimuli, suggesting the observed differences apply only to the subconscious pathway.

The Role of Emotional Content and Perceptual Saliency During the Programming of Saccades Toward Faces

Previous studies have shown that the human visual system can detect a face and elicit a saccadic eye movement toward it very efficiently compared to other categories of visual stimuli. In the first experiment, we tested the influence of facial expressions on fast face detection using a saccadic choice task. Face-vehicle pairs were simultaneously presented and participants were asked to saccade toward the target (the face or the vehicle). We observed that saccades toward faces were initiated faster, and more often in the correct direction, than saccades toward vehicles, regardless of the facial expressions (happy, fearful, or neutral). We also observed that saccade endpoints on face images were lower when the face was happy and higher when it was neutral. In the second experiment, we explicitly tested the detection of facial expressions. We used a saccadic choice task with emotional-neutral pairs of faces and participants were asked to saccade toward the emotional (happy or fearful) or the neutral face. Participants were faster when they were asked to saccade toward the emotional face. They also made fewer errors, especially when the emotional face was happy. Using computational modeling, we showed that this happy face advantage can, at least partly, be explained by perceptual factors. Also, saccade endpoints were lower when the target was happy than when it was fearful. Overall, we suggest that there is no automatic prioritization of emotional faces, at least for saccades with short latencies, but that salient local face features can automatically attract attention.

Impaired social perception from eyes and face visual cues: evidence from prefrontal cortex damage

Despite the key role that decoding of social-perceptual cues from faces plays in interpersonal communication, it is only recently that the potential of prefrontal cortex damage to disrupt this ability has been recognized. In fact, few studies to date had assessed whether the ability to identify the state of mind of others from the whole or part of the face is disrupted after prefrontal cortex damage and whether these two abilities are associated and share overlapped neural systems. In the present study, 30 patients with focal prefrontal lesions and 30 matched control subjects were assessed on their ability to recognize six basic emotions from facial expressions of the whole face and to identify states of mind of others from photographs of only the eyes using the "Reading the Mind in the Eyes Task". Results showed that frontal patients were significantly impaired compared with control subjects on both tasks. Moreover, regression analyses showed that these two abilities are associated and reciprocally predictive of one another. Finally, using voxel-based lesion analysis; we identified a partially common bilaterally distributed prefrontal network in the decoding of both emotional cues from both the whole face and eyes centered within the dorsomedial and ventral regions with extension to the lateral frontal pole.

The Influence of Key Facial Features on Recognition of Emotion in Cartoon Faces

Cartoon faces are widely used in social media, animation production, and social robots because of their attractive ability to convey different emotional information. Despite their popular applications, the mechanisms of recognizing emotional expressions in cartoon faces are still unclear. Therefore, three experiments were conducted in this study to systematically explore a recognition process for emotional cartoon expressions (happy, sad, and neutral) and to examine the influence of key facial features (mouth, eyes, and eyebrows) on emotion recognition. Across the experiments, three presentation conditions were employed: (1) a full face; (2) individual feature only (with two other features concealed); and (3) one feature concealed with two other features presented. The cartoon face images used in this study were converted from a set of real faces acted by Chinese posers, and the observers were Chinese. The results show that happy cartoon expressions were recognized more accurately than neutral and sad expressions, which was consistent with the happiness recognition advantage revealed in real face studies. Compared with real facial expressions, sad cartoon expressions were perceived as sadder, and happy cartoon expressions were perceived as less happy, regardless of whether full-face or single facial features were viewed. For cartoon faces, the mouth was demonstrated to be a feature that is sufficient and necessary for the recognition of happiness, and the eyebrows were sufficient and necessary for the recognition of sadness. This study helps to clarify the perception mechanism underlying emotion recognition in cartoon faces and sheds some light on directions for future research on intelligent human-computer interactions.

Facial expressions can be categorized along the upper-lower facial axis, from a perceptual perspective

A critical question, fundamental for building models of emotion, is how to categorize emotions. Previous studies have typically taken one of two approaches: (a) they focused on the pre-perceptual visual cues, how salient facial features or configurations were displayed; or (b) they focused on the post-perceptual affective experiences, how emotions affected behavior. In this study, we attempted to group emotions at a peri-perceptual processing level: it is well known that humans perceive different facial expressions differently, therefore, can we classify facial expressions into distinct categories in terms of their perceptual similarities? Here, using a novel non-lexical paradigm, we assessed the perceptual dissimilarities between 20 facial expressions using reaction times. Multidimensional-scaling analysis revealed that facial expressions were organized predominantly along the upper-lower face axis. Cluster analysis of behavioral data delineated three superordinate categories, and eye-tracking measurements validated these clustering results. Interestingly, these superordinate categories can be conceptualized according to how facial displays interact with acoustic communications: One group comprises expressions that have salient mouth features. They likely link to species-specific vocalization, for example, crying, laughing. The second group comprises visual displays with diagnosing features in both the mouth and the eye regions. They are not directly articulable but can be expressed prosodically, for example, sad, angry. Expressions in the third group are also whole-face expressions but are completely independent of vocalization, and likely being blends of two or more elementary expressions. We propose a theoretical framework to interpret the tripartite division in which distinct expression subsets are interpreted as successive phases in an evolutionary chain.

Effects of low-level visual information and perceptual load on P1 and N170 responses to emotional expressions

Emotional facial expressions lead to modulations of early event-related potentials (ERPs). However, it has so far remained unclear how far these modulations represent face-specific effects rather than differences in low-level visual features, and to which extent they depend on available processing resources. To examine these questions, we conducted two preregistered independent experiments (N = 40 in each experiment) using different variants of a novel task that manipulates peripheral perceptual load across levels but keeps overall visual stimulation constant. At the display center, we presented task-irrelevant angry, neutral, and happy faces and their Fourier phase-scrambled versions, which preserved low-level visual features. The results of both studies showed load-independent P1 and N170 emotional expression effects. Importantly, by using Bayesian analyses we could confirm that these facial expression effects were face-independent for the P1 but not for the N170 component. We conclude that firstly, ERP modulations during the P1 interval strongly depend on low-level visual information, while the N170 modulation requires the processing of figural facial expression features. Secondly, both P1 and N170 modulations appear to be immune to a large range of variations in perceptual load.

Children's emotion inferences from masked faces: Implications for social interactions during COVID-19

To slow the progression of COVID-19, the Centers for Disease Control (CDC) and the World Health Organization (WHO) have recommended wearing face coverings. However, very little is known about how occluding parts of the face might impact the emotion inferences that children make during social interactions. The current study recruited a racially diverse sample of school-aged (7- to 13-years) children from publicly funded after-school programs. Children made inferences from facial configurations that were not covered, wearing sunglasses to occlude the eyes, or wearing surgical masks to occlude the mouth. Children were still able to make accurate inferences about emotions, even when parts of the faces were covered. These data suggest that while there may be some challenges for children incurred by others wearing masks, in combination with other contextual cues, masks are unlikely to dramatically impair children's social interactions in their everyday lives.

Flexible face processing: Holistic processing of facial identity is modulated by task-irrelevant facial expression

Holistic processing is a hallmark of face perception and has been observed separately for facial identity and expression. While the identity of a face remains constant regardless of any changes in facial expressions, to what extent is holistic processing of facial identity affected by task-irrelevant facial expressions? If holistic processing is flexible and integrates both identity and expression information, the magnitude of holistic processing of facial identity may be systematically modulated by different facial expressions, due to either visual or emotional differences among the expressions. In Experiment 1, participants matched the identities of target halves of two sequentially presented face composites, with both composites showing either positive (happy) or negative (angry) expressions. The presentation duration for the test composite was either short (200 ms) or long (until a response). With a short presentation duration, the magnitude of holistic processing of identity for happy and angry composites was comparable. In contrast, with a long presentation duration, holistic processing of identity was reduced for angry compared with happy face composites. Experiment 2 replicated the results and showed reduced holistic processing of identity for face composites with either angry or neutral expressions, compared with happy expressions, given a long presentation duration. Because the modulation of facial expressions on holistic processing of facial identity was observed with long, but not short, presentation durations, these results suggest that such influence is unlikely solely due to visual differences, but may instead arise from cognitive evaluation of the emotions conveyed by facial expressions.

Attentional conditions differentially affect early, intermediate and late neural responses to fearful and neutral faces

The processing of fearful facial expressions is prioritized by the human brain. This priority is maintained across various information processing stages as evident in early, intermediate and late components of event-related potentials (ERPs). However, emotional modulations are inconsistently reported for these different processing stages. In this pre-registered study, we investigated how feature-based attention differentially affects ERPs to fearful and neutral faces in 40 participants. The tasks required the participants to discriminate either the orientation of lines overlaid onto the face, the sex of the face or the face's emotional expression, increasing attention to emotion-related features. We found main effects of emotion for the N170, early posterior negativity (EPN) and late positive potential (LPP). While N170 emotional modulations were task-independent, interactions of emotion and task were observed for the EPN and LPP. While EPN emotion effects were found in the sex and emotion tasks, the LPP emotion effect was mainly driven by the emotion task. This study shows that early responses to fearful faces are task-independent (N170) and likely based on low-level and configural information while during later processing stages, attention to the face (EPN) or-more specifically-to the face's emotional expression (LPP) is crucial for reliable amplified processing of emotional faces.

Implicit race attitudes modulate visual information extraction for trustworthiness judgments

Black people are still considered to be one of the most stigmatized groups and have to face multiple prejudices that undermine their well-being. Assumptions and beliefs about other racial groups are quite pervasive and have been shown to impact basic social tasks such as face processing. For example, individuals with high racial prejudice conceptualize other-race faces as less trustworthy and more criminal. However, it is unknown if implicit racial bias could modulate even low-level perceptual mechanisms such as spatial frequency (SF) extraction when judging the level of trustworthiness of other-race faces. The present study showed that although similar facial features are used to judge the trustworthiness of White and Black faces, own-race faces are processed in lower SF (i.e. coarse information such as the contour of the face and blurred shapes as opposed to high SF representing fine-grained information such as eyelashes or fine wrinkles). This pattern was modulated by implicit race biases: higher implicit biases are associated with a significantly higher reliance on low SF with White than with Black faces.

Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: an eye-tracking study

Individuals with insomnia have been found to have disturbed perception of facial expressions. Through eye movement examinations, here we test the hypothesis that this effect is due to impaired visual attention functions for retrieving diagnostic features in facial expression judgments. Twenty-three individuals with insomnia symptoms and 23 controls without insomnia completed a task to categorize happy, sad, fearful, and angry facial expressions. The participants with insomnia were less accurate in recognizing angry faces and misidentified them as fearful faces more often than the controls. A hidden Markov modeling approach for eye movement data analysis revealed that when viewing facial expressions, more individuals with insomnia adopted a nose-mouth eye movement pattern focusing on the vertical face midline while more controls adopted an eyes-mouth pattern preferentially attending to lateral features, particularly the two eyes. As previous studies found that the primary diagnostic feature for recognizing angry faces is the eyes while the diagnostic features for other facial expressions involve the mouth region, missing the eye region may contribute to specific difficulties in recognizing angry facial expressions, consistent with our behavioral finding in participants with insomnia symptoms. Taken together, the findings suggest that impaired information selection through visual attention control may be related to the compromised emotion perception in individuals with insomnia.

Socially anxious individuals perform better using low spatial frequency information to process facial expressions and objects

BACKGROUND AND OBJECTIVES

It is well known that socially anxious individuals show biased processing towards threatening faces and better performance with low spatial frequencies information (LSF). It is unclear whether this bias is confined to facial processing or can be extended to other types of information.

METHODS

Two experimental phases involving discrimination tasks considering neutral and angry facial expressions and everyday objects in two different spatial frequency conditions were conducted to compare the performance of "socially and non-socially anxious individuals".

RESULTS

Findings showed that highly socially anxious individuals (HSAi) were faster in decision processing for LSF neutral faces than LSF angry faces and responded more slowly to LSF angry faces than unfiltered angry faces. Moreover, they responded more quickly to LSF object images than low socially anxious individuals (LSAi).

LIMITATIONS

The fact that the participants were not diagnosed with social anxiety disorder limits the relevance of clinical findings. The study is further limited because it compared and contrasted only two emotional expressions and two frequency bands.

CONCLUSIONS

Study results showed that HSAi better process LSF neutral information and that this advantage is not limited to neutral faces alone, but extends across other domains.

Effects of male postpartum depression on father-infant interaction: The mediating role of face processing

It is estimated that postpartum depression affects up to 25% of men. Despite such high prevalence, the majority of studies on postpartum depression are focused on mothers, and the role of paternal depression and its effects on infant development have been overlooked by researchers and clinicians. The present study aimed to fill this gap by investigating the effect of paternal postpartum depression on father-infant interactions. In addition, we examined whether differences in face recognition mediated the effects of paternal postpartum depression on father-infant interactions. A total of 61 father-infant dyads (17 postpartum depression, 44 controls) took part in the study. Results revealed that compared to controls, fathers with postpartum depression had a worse pattern of interaction with their infants on measures of responsiveness, mood, and sensitivity; they also had greater difficulty in recognizing happy adult faces, but greater facility in recognizing sad adult faces. Depressed fathers attributed greater intensities to sad adult and infant faces. The tendency to attribute greater intensity to sad adult faces was confirmed as a partial mediator of the effect of paternal postpartum depression on measures of father responsiveness and as a full mediator of the effects of paternal depression on father sensitivity. Clinical implications and suggestions for further studies are discussed.

Dynamic Construction of Reduced Representations in the Brain for Perceptual Decision Behavior

Over the past decade, extensive studies of the brain regions that support face, object, and scene recognition suggest that these regions have a hierarchically organized architecture that spans the occipital and temporal lobes [1-14], where visual categorizations unfold over the first 250 ms of processing [15-19]. This same architecture is flexibly involved in multiple tasks that require task-specific representations-e.g. categorizing the same object as "a car" or "a Porsche." While we partly understand where and when these categorizations happen in the occipito-ventral pathway, the next challenge is to unravel how these categorizations happen. That is, how does high-dimensional input collapse in the occipito-ventral pathway to become low dimensional representations that guide behavior? To address this, we investigated what information the brain processes in a visual perception task and visualized the dynamic representation of this information in brain activity. To do so, we developed stimulus information representation (SIR), an information theoretic framework, to tease apart stimulus information that supports behavior from that which does not. We then tracked the dynamic representations of both in magneto-encephalographic (MEG) activity. Using SIR, we demonstrate that a rapid (∼170 ms) reduction of behaviorally irrelevant information occurs in the occipital cortex and that representations of the information that supports distinct behaviors are constructed in the right fusiform gyrus (rFG). Our results thus highlight how SIR can be used to investigate the component processes of the brain by considering interactions between three variables (stimulus information, brain activity, behavior), rather than just two, as is the current norm.

Development of the Youth Emotion Picture Set

Several facial expression image sets have been developed. Nevertheless, there is a lack of facial expression sets comprising adolescents’ images depicting all basic emotions. This study aimed to fill this gap through the development of an image database of youth facial expressions, containing pictures of six basic emotions plus neutral. Posed and spontaneous expressions were collected from 31 youths, 12 to 20 years old; 2,279 frames were obtained, and an initial screening was conducted through the exclusion of similar frames, low intensity images, and ambiguous or blended expressions; 256 frames met criteria and were rechecked by two expert judges. Images were retained if they depicted all the prototypical features of the designated expression. A final selection was conducted to assure an image set that covered all age ranges, both sexes and an even number of images by expression, resulting in 42 frames (21 male, six of each emotion). Expert judges, independent adults, independent teenagers, and a software validation were used to assure database validity. Agreement across raters was high, and no differences were observed for posed and spontaneous images. The data set developed in this study can be a valid tool in studies of facial expressions, in particular, with adolescents’ samples.

Effects of Briefly Presented Masked Emotional Facial Expressions on Gaze Behavior: An Eye-Tracking Study

Orientation of gaze toward specific regions of the face such as the eyes or the mouth helps to correctly identify the underlying emotion. The present eye-tracking study investigates whether facial features diagnostic of specific emotional facial expressions are processed preferentially, even when presented outside of subjective awareness. Eye movements of 73 healthy individuals were recorded while completing an affective priming task. Primes (pictures of happy, neutral, sad, angry, and fearful facial expressions) were presented for 50 ms with forward and backward masking. Participants had to evaluate subsequently presented neutral faces. Results of an awareness check indicated that participants were subjectively unaware of the emotional primes. No affective priming effects were observed but briefly presented emotional facial expressions elicited early eye movements toward diagnostic regions of the face. Participants oriented their gaze more rapidly to the eye region of the neutral mask after a fearful facial expression. After a happy facial expression, participants oriented their gaze more rapidly to the mouth region of the neutral mask. Moreover, participants dwelled longest on the eye region after a fearful facial expression, and the dwell time on the mouth region was longest for happy facial expressions. Our findings support the idea that briefly presented fearful and happy facial expressions trigger an automatic mechanism that is sensitive to the distribution of relevant facial features and facilitates the orientation of gaze toward them.

Identifying and detecting facial expressions of emotion in peripheral vision

Facial expressions of emotion are signals of high biological value. Whilst recognition of facial expressions has been much studied in central vision, the ability to perceive these signals in peripheral vision has only seen limited research to date, despite the potential adaptive advantages of such perception. In the present experiment, we investigate facial expression recognition and detection performance for each of the basic emotions (plus neutral) at up to 30 degrees of eccentricity. We demonstrate, as expected, a decrease in recognition and detection performance with increasing eccentricity, with happiness and surprised being the best recognized expressions in peripheral vision. In detection however, while happiness and surprised are still well detected, fear is also a well detected expression. We show that fear is a better detected than recognized expression. Our results demonstrate that task constraints shape the perception of expression in peripheral vision and provide novel evidence that detection and recognition rely on partially separate underlying mechanisms, with the latter more dependent on the higher spatial frequency content of the face stimulus.

Atypical information-use in children with autism spectrum disorder during judgments of child and adult face identity

Unusual patterns of fixation behavior in individuals with autism spectrum disorder during face tasks hint at atypical processing strategies that could contribute to diminished face expertise in this group. Here, we use the Bubbles reverse correlation technique to directly examine face-processing strategies during identity judgments in children with and without autism, and typical adults. Results support a qualitative atypicality in autistic face processing. We identify clear differences not only in the specific features relied upon for face judgments, but also more generally in the extent to which they demonstrate a flexible and adaptive profile of information use.

Automatic Facial Expression Recognition System Using Deep Network-Based Data Fusion

This paper presents a novel automatic facial expressions recognition system (AFERS) using the deep network framework. The proposed AFERS consists of four steps: 1) geometric features extraction; 2) regional local binary pattern (LBP) features extraction; 3) fusion of both the features using autoencoders; and 4) classification using Kohonen self-organizing map (SOM)-based classifier. This paper makes three distinct contributions. The proposed deep network consisting of autoencoders and the SOM-based classifier is computationally more efficient and performance wise more accurate. The fusion of geometric features with LBP features using autoencoders provides better representation of facial expression. The SOM-based classifier proposed in this paper has been improved by making use of a soft-threshold logic and a better learning algorithm. The performance of the proposed approach is validated on two widely used databases (DBs): 1) MMI and 2) extended Cohn-Kanade (CK+). An average recognition accuracy of 97.55% in MMI DB and 98.95% in CK+ DB are obtained using the proposed algorithm. The recognition results obtained from fused features are found to be distinctly superior to both recognition using individual features as well as recognition with a direct concatenation of the individual feature vectors. Simulation results validate that the proposed AFERS is more efficient as compared to the existing approaches.

Behavioral and Neuroimaging Evidence for Facial Emotion Recognition in Elderly Korean Adults with Mild Cognitive Impairment, Alzheimer's Disease, and Frontotemporal Dementia

Background: Facial emotion recognition (FER) is impaired in individuals with frontotemporal dementia (FTD) and Alzheimer’s disease (AD) when compared to healthy older adults. Since deficits in emotion recognition are closely related to caregiver burden or social interactions, researchers have fundamental interest in FER performance in patients with dementia.

Purpose: The purpose of this study was to identify the performance profiles of six facial emotions (i.e., fear, anger, disgust, sadness, surprise, and happiness) and neutral faces measured among Korean healthy control (HCs), and those with mild cognitive impairment (MCI), AD, and FTD. Additionally, the neuroanatomical correlates of facial emotions were investigated.

Methods: A total of 110 (33 HC, 32 MCI, 32 AD, 13 FTD) older adult participants were recruited from two different medical centers in metropolitan areas of South Korea. These individuals underwent an FER test that was used to assess the recognition of emotions or absence of emotion (neutral) in 35 facial stimuli. Repeated measures two-way analyses of variance were used to examine the distinct profiles of emotional recognition among the four groups. We also performed brain imaging and voxel-based morphometry (VBM) on the participants to examine the associations between FER scores and gray matter volume.

Results: The mean score of negative emotion recognition (i.e., fear, anger, disgust, and sadness) clearly discriminated FTD participants from individuals with MCI and AD and HC [F(3,106) = 10.829, p < 0.001, η² = 0.235], whereas the mean score of positive emotion recognition (i.e., surprise and happiness) did not. A VBM analysis showed negative emotions were correlated with gray matter volume of anterior temporal regions, whereas positive emotions were related to gray matter volume of fronto-parietal regions.

Conclusion: Impairment of negative FER in patients with FTD is cross-cultural. The discrete neural correlates of FER indicate that emotional recognition processing is a multi-modal system in the brain. Focusing on the negative emotion recognition is a more effective way to discriminate healthy aging, MCI, and AD from FTD in older Korean adults.

A cognição social dos psicopatas: achados científicos recentes

Resumo O psicopata apresenta traços na personalidade relacionados à ausência de remorso e uma maior dominância social, expressos muitas vezes como manipulação de outros indivíduos. Em estudos atuais sugere-se que psicopatas podem apresentar deficiências no processamento de estímulos emocionais em uma situação de interação social. Este estudo tem por objetivo realizar uma revisão teórica, não sistemática, discutindo pesquisas recentes sobre o tema. A partir da leitura, conclui-se que novas considerações revelam-se pertinentes, pois nem sempre o tipo de disfunção da cognição social dos psicopatas é explícito na literatura. Sugere-se que outros métodos para avaliar a capacidade dos psicopatas em identificar as emoções possam ser investigadas, além de identificar a medida das estratégias interpessoais. Em termos gerais, apresenta-se uma proposta de reflexão para um transtorno cuja compreensão etiológica deve ser biopsicossocial, contribuindo para assinalar novas direções nas pesquisas voltadas para a cognição social dos psicopatas.

Oxytocin Reduces Face Processing Time but Leaves Recognition Accuracy and Eye-Gaze Unaffected

OBJECTIVES

Previous studies have found that oxytocin (OXT) can improve the recognition of emotional facial expressions; it has been proposed that this effect is mediated by an increase in attention to the eye-region of faces. Nevertheless, evidence in support of this claim is inconsistent, and few studies have directly tested the effect of oxytocin on emotion recognition via altered eye-gaze Methods: In a double-blind, within-subjects, randomized control experiment, 40 healthy male participants received 24 IU intranasal OXT and placebo in two identical experimental sessions separated by a 2-week interval. Visual attention to the eye-region was assessed on both occasions while participants completed a static facial emotion recognition task using medium intensity facial expressions.

RESULTS

Although OXT had no effect on emotion recognition accuracy, recognition performance was improved because face processing was faster across emotions under the influence of OXT. This effect was marginally significant (p<.06). Consistent with a previous study using dynamic stimuli, OXT had no effect on eye-gaze patterns when viewing static emotional faces and this was not related to recognition accuracy or face processing time.

CONCLUSIONS

These findings suggest that OXT-induced enhanced facial emotion recognition is not necessarily mediated by an increase in attention to the eye-region of faces, as previously assumed. We discuss several methodological issues which may explain discrepant findings and suggest the effect of OXT on visual attention may differ depending on task requirements. (JINS, 2017, 23, 23-33).

Identity information content depends on the type of facial movement

Facial movements convey information about many social cues, including identity. However, how much information about a person's identity is conveyed by different kinds of facial movements is unknown. We addressed this question using a recent motion capture and animation system, with which we animated one avatar head with facial movements of three types: (1) emotional, (2) emotional in social interaction and (3) conversational, all recorded from several actors. In a delayed match-to-sample task, observers were best at matching actor identity across conversational movements, worse with emotional movements in social interactions, and at chance level with emotional facial expressions. Model observers performing this task showed similar performance profiles, indicating that performance variation was due to differences in information content, rather than processing. Our results suggest that conversational facial movements transmit more dynamic identity information than emotional facial expressions, thus suggesting different functional roles and processing mechanisms for different types of facial motion.

Neural processing of fearful and happy facial expressions during emotion-relevant and emotion-irrelevant tasks: A fixation-to-feature approach

Research suggests an important role of the eyes and mouth for discriminating facial expressions of emotion. A gaze-contingent procedure was used to test the impact of fixation to facial features on the neural response to fearful, happy and neutral facial expressions in an emotion discrimination (Exp.1) and an oddball detection (Exp.2) task. The N170 was the only eye-sensitive ERP component, and this sensitivity did not vary across facial expressions. In both tasks, compared to neutral faces, responses to happy expressions were seen as early as 100-120ms occipitally, while responses to fearful expressions started around 150ms, on or after the N170, at both occipital and lateral-posterior sites. Analyses of scalp topographies revealed different distributions of these two emotion effects across most of the epoch. Emotion processing interacted with fixation location at different times between tasks. Results suggest a role of both the eyes and mouth in the neural processing of fearful expressions and of the mouth in the processing of happy expressions, before 350ms.

Attentional bias towards and away from fearful faces is modulated by developmental amygdala damage

The amygdala is believed to play a major role in orienting attention towards threat-related stimuli. However, behavioral studies on amygdala-damaged patients have given inconsistent results-variously reporting decreased, persisted, and increased attention towards threat. Here we aimed to characterize the impact of developmental amygdala damage on emotion perception and the nature and time-course of spatial attentional bias towards fearful faces. We investigated SF, a 14-year-old with selective bilateral amygdala damage due to Urbach-Wiethe disease (UWD), and ten healthy controls. Participants completed a fear sensitivity questionnaire, facial expression classification task, and dot-probe task with fearful or neutral faces for spatial cueing. Three cue durations were used to assess the time-course of attentional bias. SF expressed significantly lower fear sensitivity, and showed a selective impairment in classifying fearful facial expressions. Despite this impairment in fear recognition, very brief (100 msec) fearful cues could orient SF's spatial attention. In healthy controls, the attentional bias emerged later and persisted longer. SF's attentional bias was due solely to facilitated engagement to fear, while controls showed the typical phenomenon of difficulty in disengaging from fear. Our study is the first to demonstrate the separable effects of amygdala damage on engagement and disengagement of spatial attention. The findings indicate that multiple mechanisms contribute in biasing attention towards fear, which vary in their timing and dependence on amygdala integrity. It seems that the amygdala is not essential for rapid attention to emotion, but probably has a role in assessment of biological relevance.

The Human Face as a Dynamic Tool for Social Communication

As a highly social species, humans frequently exchange social information to support almost all facets of life. One of the richest and most powerful tools in social communication is the face, from which observers can quickly and easily make a number of inferences - about identity, gender, sex, age, race, ethnicity, sexual orientation, physical health, attractiveness, emotional state, personality traits, pain or physical pleasure, deception, and even social status. With the advent of the digital economy, increasing globalization and cultural integration, understanding precisely which face information supports social communication and which produces misunderstanding is central to the evolving needs of modern society (for example, in the design of socially interactive digital avatars and companion robots). Doing so is challenging, however, because the face can be thought of as comprising a high-dimensional, dynamic information space, and this impacts cognitive science and neuroimaging, and their broader applications in the digital economy. New opportunities to address this challenge are arising from the development of new methods and technologies, coupled with the emergence of a modern scientific culture that embraces cross-disciplinary approaches. Here, we briefly review one such approach that combines state-of-the-art computer graphics, psychophysics and vision science, cultural psychology and social cognition, and highlight the main knowledge advances it has generated. In the light of current developments, we provide a vision of the future directions in the field of human facial communication within and across cultures.

Orienting to face expression during encoding improves men's recognition of own gender faces

It is unclear why women have superior episodic memory of faces, but the benefit may be partially the result of women engaging in superior processing of facial expressions. Therefore, we hypothesized that orienting instructions to attend to facial expression at encoding would significantly improve men's memory of faces and possibly reduce gender differences. We directed 203 college students (122 women) to study 120 faces under instructions to orient to either the person's gender or their emotional expression. They later took a recognition test of these faces by either judging whether they had previously studied the same person or that person with the exact same expression; the latter test evaluated recollection of specific facial details. Orienting to facial expressions during encoding significantly improved men's recognition of own-gender faces and eliminated the advantage that women had for male faces under gender orienting instructions. Although gender differences in spontaneous strategy use when orienting to faces cannot fully account for gender differences in face recognition, orienting men to facial expression during encoding is one way to significantly improve their episodic memory for male faces.

Fixation to features and neural processing of facial expressions in a gender discrimination task

Early face encoding, as reflected by the N170 ERP component, is sensitive to fixation to the eyes. Whether this sensitivity varies with facial expressions of emotion and can also be seen on other ERP components such as P1 and EPN, was investigated. Using eye-tracking to manipulate fixation on facial features, we found the N170 to be the only eye-sensitive component and this was true for fearful, happy and neutral faces. A different effect of fixation to features was seen for the earlier P1 that likely reflected general sensitivity to face position. An early effect of emotion (∼120 ms) for happy faces was seen at occipital sites and was sustained until ∼350 ms post-stimulus. For fearful faces, an early effect was seen around 80 ms followed by a later effect appearing at ∼150 ms until ∼300 ms at lateral posterior sites. Results suggests that in this emotion-irrelevant gender discrimination task, processing of fearful and happy expressions occurred early and largely independently of the eye-sensitivity indexed by the N170. Processing of the two emotions involved different underlying brain networks active at different times.

The independence of expression and identity in face-processing: evidence from neuropsychological case studies

The processing of facial identity and facial expression have traditionally been seen as independent—a hypothesis that has largely been informed by a key double dissociation between neurological patients with a deficit in facial identity recognition but not facial expression recognition, and those with the reverse pattern of impairment. The independence hypothesis is also reflected in more recent anatomical models of face-processing, although these theories permit some interaction between the two processes. Given that much of the traditional patient-based evidence has been criticized, a review of more recent case reports that are accompanied by neuroimaging data is timely. Further, the performance of individuals with developmental face-processing deficits has recently been considered with regard to the independence debate. This paper reviews evidence from both acquired and developmental disorders, identifying methodological and theoretical strengths and caveats in these reports, and highlighting pertinent avenues for future research.

Anatomical curve identification

Methods for capturing images in three dimensions are now widely available, with stereo-photogrammetry and laser scanning being two common approaches. In anatomical studies, a number of landmarks are usually identified manually from each of these images and these form the basis of subsequent statistical analysis. However, landmarks express only a very small proportion of the information available from the images. Anatomically defined curves have the advantage of providing a much richer expression of shape. This is explored in the context of identifying the boundary of breasts from an image of the female torso and the boundary of the lips from a facial image. The curves of interest are characterised by ridges or valleys. Key issues in estimation are the ability to navigate across the anatomical surface in three-dimensions, the ability to recognise the relevant boundary and the need to assess the evidence for the presence of the surface feature of interest. The first issue is addressed by the use of principal curves, as an extension of principal components, the second by suitable assessment of curvature and the third by change-point detection. P-spline smoothing is used as an integral part of the methods but adaptations are made to the specific anatomical features of interest. After estimation of the boundary curves, the intermediate surfaces of the anatomical feature of interest can be characterised by surface interpolation. This allows shape variation to be explored using standard methods such as principal components. These tools are applied to a collection of images of women where one breast has been reconstructed after mastectomy and where interest lies in shape differences between the reconstructed and unreconstructed breasts. They are also applied to a collection of lip images where possible differences in shape between males and females are of interest.

Mechanisms of object recognition: what we have learned from pigeons

Behavioral studies of object recognition in pigeons have been conducted for 50 years, yielding a large body of data. Recent work has been directed toward synthesizing this evidence and understanding the visual, associative, and cognitive mechanisms that are involved. The outcome is that pigeons are likely to be the non-primate species for which the computational mechanisms of object recognition are best understood. Here, we review this research and suggest that a core set of mechanisms for object recognition might be present in all vertebrates, including pigeons and people, making pigeons an excellent candidate model to study the neural mechanisms of object recognition. Behavioral and computational evidence suggests that error-driven learning participates in object category learning by pigeons and people, and recent neuroscientific research suggests that the basal ganglia, which are homologous in these species, may implement error-driven learning of stimulus-response associations. Furthermore, learning of abstract category representations can be observed in pigeons and other vertebrates. Finally, there is evidence that feedforward visual processing, a central mechanism in models of object recognition in the primate ventral stream, plays a role in object recognition by pigeons. We also highlight differences between pigeons and people in object recognition abilities, and propose candidate adaptive specializations which may explain them, such as holistic face processing and rule-based category learning in primates. From a modern comparative perspective, such specializations are to be expected regardless of the model species under study. The fact that we have a good idea of which aspects of object recognition differ in people and pigeons should be seen as an advantage over other animal models. From this perspective, we suggest that there is much to learn about human object recognition from studying the "simple" brains of pigeons.