Facing mixed emotions: Analytic and holistic perception of facial emotion expressions engages separate brain networks

Binocular rivalry reveals differential face processing in congenital prosopagnosia

Congenital Prosopagnosia (CP) is an innate impairment in face perception with heterogeneous characteristics. It is still unclear if and to what degree holistic processing of faces is disrupted in CP. Such disruption would be expected to lead to a focus on local features of the face. In this study, we used binocular rivalry (BR) to implicitly measure face perception in conditions that favour holistic or local processing. The underlying assumption is that if stimulus saliency affects the perceptual dominance of a given stimulus in BR, one can deduce how salient a stimulus is for a given group (here: participants with and without CP) based on the measured perceptual dominance. A further open question is whether the deficit in face processing in CP extends to the processing of the facial display of emotions. In experiment 1, we compared predominance of upright and inverted faces displaying different emotions (fearful, happy, neutral) vs. houses between participants with CP (N = 21) and with normal face perception (N = 21). The results suggest that CP observers process emotions in faces automatically but rely more on local features than controls. The inversion of faces, which is supposed to disturb holistic processing, affected controls in a more pronounced way than participants with CP. In experiment 2, we introduced the Thatcher effect in BR by inverting the eye and mouth regions of the presented faces in the hope of further increasing the effect of face inversion. However, our expectations were not borne out by the results. Critically, both experiments showed that inversion effects were more pronounced in controls than in CP, suggesting that holistic face processing is less relevant in CP. We find BR to be a useful implicit test for assessing visual processing specificities in neurological participants.

No influence of emotional expression on size underestimation of upright faces

Faces are a primary means of conveying social information between humans. One important factor modulating the perception of human faces is emotional expression. Face inversion also affects perception, including judgments of emotional expression, possibly through the disruption of configural processing. One intriguing inversion effect is an illusion whereby faces appear to be physically smaller when upright than when inverted. This illusion appears to be highly selective for faces. In this study, we investigated whether the emotional expression of a face (neutral, happy, afraid, and angry) modulates the magnitude of this size illusion. Results showed that for all four expressions, there was a clear bias for inverted stimuli to be judged as larger than upright ones. This demonstrates that there is no influence of emotional expression on the size underestimation of upright faces, a surprising result given that recognition of different emotional expressions is known to be affected unevenly by inversion. Results are discussed considering recent neuroimaging research which used population receptive field (pRF) mapping to investigate the neural mechanisms underlying face perception features and which may provide an explanation for how an upright face appears smaller than an inverted one. Elucidation of this effect would lead to a greater understanding of how humans communicate.

To see or not to see: the parallel processing of self-relevance and facial expressions

The self, like the concept of central "gravity", facilitates the processing of information that is directly relevant to the self. This phenomenon is known as the self-prioritization effect. However, it remains unclear whether the self-prioritization effect extends to the processing of emotional facial expressions. To fill this gap, we used a self-association paradigm to investigate the impact of self-relevance on the recognition of emotional facial expressions while controlling for confounding factors such as familiarity and overlearning. Using a large and diverse sample, we replicated the effect of self-relevance on face processing but found no evidence for a modulation of self-relevance on facial emotion recognition. We propose two potential theoretical explanations to account for these findings and emphasize that further research with different experimental designs and a multitasks measurement approach is needed to understand this mechanism fully. Overall, our study contributes to the literature on the parallel cognitive processing of self-relevance and facial emotion recognition, with implications for both social and cognitive psychology.

Analytic-holistic cognitive styles affect consumer responses to food and beverage samples during sensory evaluation

Recent studies have shown that the analytic-holistic theory is applicable in sensory science-related areas. Analytic and holistic cognitive style groups have been found to have significantly different perceptions and behaviors within food-related scenarios. These differences were further investigated and identified within the current study, focusing on specific areas of common sensory tests and analyses where analytic and holistic cognitive style groups may differ from one another. Before the main study, 419 volunteers were classified into three groups based on their scores to the Analysis-Holism Scale (AHS). The extreme groups (65 adults for each) were identified as the "analytic" and "holistic" cognitive style groups, respectively. Participants evaluated fruit-flavored beverages and fruit samples for their impression of the intensity or hedonic aspects in the study conducted over two sessions. Each session either employed solely category or line scale for the questions. Analyses focused on the analytic and holistic group comparisons in mean, variance, penalty analysis, and pre- and post-AHS score differences. Results showed that the holistic group exhibited significantly higher mean scores and standard deviations in the hedonic ratings of fruit samples than the analytic group did. Compared to the analytic group, the holistic group showed significantly smaller mean drops in overall liking across the five Just-About-Right (JAR) questions related to flavors or tastes of the mixed-fruit flavored water. A significant difference between the AHS scores measured before and after the sensory evaluation was observed in the holistic group, but not in the analytic group. In conclusion, our findings provide empirical evidence that cognitive styles affect consumer responses to food or beverage samples during sensory evaluation tasks. This implies that sensory professionals may consider these analytic-holistic contrasts while exploring consumer responses to their target samples.

Development and validation of a food-related analysis-holism scale (F-AHS)

Recent studies have indicated analytic-holistic cognitive style differences of consumers can significantly impact perceptions, opinions, and behaviors toward foods. Interestingly, these studies have also offered evidence that the sole measurement tool to assess analytic-holistic tendencies, the analysis-holism scale (AHS), may not accurately apply to food-experience-related research scenarios. Due to these notions, Studies 1 and 2 employed the use of 465 and 487 participants, respectively, to develop, refine, and finalize a food-related AHS (F-AHS) using exploratory and confirmatory factor analyses and prior research on analytic-holistic differences and scale development. Study 3 was conducted to validate the newly developed F-AHS, based on the results of Studies 1 and 2, by replicating the procedures and analyses from prior research while using the F-AHS instead of the AHS to segment participants (N = 130). The results of Study 3 provided consistent evidence that the F-AHS better separated participants into analytic and holistic groups than the AHS through larger analytic-holistic differences that more closely aligned with prior analytic-holistic research. Our findings showed that the F-AHS is capable of separating consumers into analytic and holistic cognitive style groups and is better suited to sensory and consumer-related applications than the AHS. PRACTICAL APPLICATION: Our findings provide researchers with an optimized tool to separate consumers into analytic and holistic groups in food-experience-related situations. The food-related analysis-holism scale can then be used in both academia and industry to separate consumers based on cognitive style to better understand how and why consumers show variations in perception, acceptance, and behavior in food-related experiences.

Emotion is perceived accurately from isolated body parts, especially hands

Body posture and configuration provide important visual cues about the emotion states of other people. We know that bodily form is processed holistically, however, emotion recognition may depend on different mechanisms; certain body parts, such as the hands, may be especially important for perceiving emotion. This study therefore compared participants' emotion recognition performance when shown images of full bodies, or of isolated hands, arms, heads and torsos. Across three experiments, emotion recognition accuracy was above chance for all body parts. While emotions were recognized most accurately from full bodies, recognition performance from the hands was more accurate than for other body parts. Representational similarity analysis further showed that the pattern of errors for the hands was related to that for full bodies. Performance was reduced when stimuli were inverted, showing a clear body inversion effect. The high performance for hands was not due only to the fact that there are two hands, as performance remained well above chance even when just one hand was shown. These results demonstrate that emotions can be decoded from body parts. Furthermore, certain features, such as the hands, are more important to emotion perception than others. STATEMENT OF RELEVANCE: Successful social interaction relies on accurately perceiving emotional information from others. Bodies provide an abundance of emotion cues; however, the way in which emotional bodies and body parts are perceived is unclear. We investigated this perceptual process by comparing emotion recognition for body parts with that for full bodies. Crucially, we found that while emotions were most accurately recognized from full bodies, emotions were also classified accurately when images of isolated hands, arms, heads and torsos were seen. Of the body parts shown, emotion recognition from the hands was most accurate. Furthermore, shared patterns of emotion classification for hands and full bodies suggested that emotion recognition mechanisms are shared for full bodies and body parts. That the hands are key to emotion perception is important evidence in its own right. It could also be applied to interventions for individuals who find it difficult to read emotions from faces and bodies.

Facial mask disturbs ocular exploration but not pupil reactivity

Introduction

The COVID-19 pandemic has imposed to wear a face mask that may have negative consequences for social interactions despite its health benefits. A lot of recent studies focused on emotion recognition of masked faces, as the mouth is, with the eyes, essential to convey emotional content. However, none have studied neurobehavioral and neurophysiological markers of masked faces perception, such as ocular exploration and pupil reactivity. The purpose of this eye tracking study was to quantify how wearing a facial accessory, and in particular a face mask, affected the ocular and pupillary response to a face, emotional or not.

Methods

We used videos of actors wearing a facial accessory to characterize the visual exploration and pupillary response in several occlusion (no accessory, sunglasses, scarf, and mask) and emotional conditions (neutral, happy, and sad) in a population of 44 adults.

Results

We showed that ocular exploration differed for face covered with an accessory, and in particular a mask, compared to the classical visual scanning pattern of a non-covered face. The covered areas of the face were less explored. Pupil reactivity seemed only slightly affected by the mask, while its sensitivity to emotions was observed even in the presence of a facial accessory.

Discussion

These results suggest a mixed impact of the mask on attentional capture and physiological adjustment, which does not seem to be reconcilable with its strong effect on behavioral emotional recognition previously described.

Integrative interaction of emotional speech in audio-visual modality

Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.

COVID-19 masks: A barrier to facial and vocal information

With the COVID-19 pandemic, we have become used to wearing masks and have experienced how masks seem to impair emotion and speech recognition. While several studies have focused on facial emotion recognition by adding images of masks on photographs of emotional faces, we have created a video database with actors really wearing masks to test its effect in more ecological conditions. After validating the emotions displayed by the actors, we found that surgical mask impaired happiness and sadness recognition but not neutrality. Moreover, for happiness, this effect was specific to the mask and not to covering the lower part of the face, possibly due to a cognitive bias associated with the surgical mask. We also created videos with speech and tested the effect of mask on emotion and speech recognition when displayed in auditory, visual, or audiovisual modalities. In visual and audiovisual modalities, mask impaired happiness and sadness but improved neutrality recognition. Mask impaired the recognition of bilabial syllables regardless of modality. In addition, it altered speech recognition only in the audiovisual modality for participants above 70 years old. Overall, COVID-19 masks mainly impair emotion recognition, except for older participants for whom it also impacts speech recognition, probably because they rely more on visual information to compensate age-related hearing loss.

Face masks affect perception of happy faces in deaf people

The SARS-CoV-2 pandemic has led significant social repercussions and forced people to wear face masks. Recent research has demonstrated that the human ability to infer emotions from facial configurations is significantly reduced when face masks are worn. Since the mouth region is specifically crucial for deaf people who speak sign language, the current study assessed the impact of face masks on inferring emotional facial expressions in a population of adult deaf signers. A group of 34 congenitally deaf individuals and 34 normal-hearing individuals were asked to identify happiness, sadness, fear, anger, and neutral expression on static human pictures with and without facial masks presented through smartphones. For each emotion, the percentage of correct responses with and without face masks was calculated and compared between groups. Results indicated that face masks, such as those worn due to the SARS-CoV-2 pandemic, limit the ability of people to infer emotions from facial expressions. The negative impact of face masks is significantly pronounced when deaf people have to recognize low-intensity expressions of happiness. These findings are of essential importance because difficulties in recognizing emotions from facial expressions due to mask wearing may contribute to the communication challenges experienced by the deaf community during the SARS-CoV-2 pandemic, generating feelings of frustration and exclusion.

Specificity in the Study of Mixed Emotions: A Theoretical Framework

Research on mixed emotions is yet to consider emotion-specificity, the idea that same-valenced emotions have distinctive characteristics and functions. We review two decades of research on mixed emotions, focusing on evidence for the occurrence of mixed emotions and the effects of mixed emotions on downstream outcomes. We then propose a novel theoretical framework of mixed-emotion-specificity with three foundational tenets: (a) Mixed emotions are distinguishable from single-valenced emotions and other mixed emotions based on their emotion-appraisal relationships; (b) Mixed emotions can further be characterized by four patterns that describe relationships between simultaneous appraisals or appraisals that are unique to mixed emotions; and (c) Carryover effects occur only on outcomes that are associated with the appraisal characteristics of mixed emotion. We outline how mixed-emotion-specific effects can be predicted based on the appraisal tendency framework. Temporal dynamics, the application of mixed-emotion-specificity to individual difference research, methodological issues, and future directions are also discussed.

Amygdala function in emotion, cognition, and behavior

The amygdala is a core structure in the anterior medial temporal lobe, with an important role in several brain functions involving memory, emotion, perception, social cognition, and even awareness. As a key brain structure for saliency detection, it triggers and controls widespread modulatory signals onto multiple areas of the brain, with a great impact on numerous aspects of adaptive behavior. Here we discuss the neural mechanisms underlying these functions, as established by animal and human research, including insights provided in both healthy and pathological conditions.

Altered Functional Connectivity Density and Couplings in Postpartum Depression with and Without Anxiety

Postpartum depression (PPD) is the most common psychological health issue among women, which often comorbids with anxiety (PPD-A). PPD and PPD-A showed highly overlapping clinical symptoms. Identifying disorder-specific neurophysiological markers of PDD and PPD-A is important for better clinical diagnosis and treatments. Here, we performed functional connectivity density (FCD) and resting-state functional connectivity (rsFC) analyses in 138 participants (45 unmedicated patients with first-episode PPD, 31 PDD-A patients and 62 healthy postnatal women, respectively). FCD mapping revealed specifically weaker long-range FCD in right lingual gyrus (LG.R) for PPD patients and significantly stronger long-range FCD in left ventral striatum (VS.L) for PPD-A patients. The follow-up rsFC analyses further revealed reduced functional connectivity between dorsomedial prefrontal cortex (dmPFC) and VS.L in both PPD and PPD-A. PPD showed specific changes of rsFC between LG.R and dmPFC, right angular gyrus and left precentral gyrus, while PPD-A represented specifically abnormal rsFC between VS.L and left ventrolateral prefrontal cortex. Moreover, the altered FCD and rsFC were closely associated with depression and anxiety symptoms load. Taken together, our study is the first to identify common and disorder-specific neural circuit disruptions in PPD and PPD-A, which may facilitate more effective diagnosis and treatments.

The rapid and automatic categorization of facial expression changes in highly variable natural images

Emotional expressions are quickly and automatically read from human faces under natural viewing conditions. Yet, categorization of facial expressions is typically measured in experimental contexts with homogenous sets of face stimuli. Here we evaluated how the 6 basic facial emotions (Fear, Disgust, Happiness, Anger, Surprise or Sadness) can be rapidly and automatically categorized with faces varying in head orientation, lighting condition, identity, gender, age, ethnic origin and background context. High-density electroencephalography was recorded in 17 participants viewing 50 s sequences with natural variable images of neutral-expression faces alternating at a 6 Hz rate. Every five stimuli (1.2 Hz), variable natural images of one of the six basic expressions were presented. Despite the wide physical variability across images, a significant F/5 = 1.2 Hz response and its harmonics (e.g., 2F/5 = 2.4 Hz, etc.) was observed for all expression changes at the group-level and in every individual participant. Facial categorization responses were found mainly over occipito-temporal sites, with distinct hemispheric lateralization and cortical topographies according to the different expressions. Specifically, a stronger response was found to Sadness categorization, especially over the left hemisphere, as compared to Fear and Happiness, together with a right hemispheric dominance for categorization of Fearful faces. Importantly, these differences were specific to upright faces, ruling out the contribution of low-level visual cues. Overall, these observations point to robust rapid and automatic facial expression categorization processes in the human brain.

Implicit evidence on the dissociation of identity and emotion recognition

Recognition of identity and of emotional facial expressions of individuals are both based on processing of the human face. While most studies show these abilities to be dissociated, some others find evidence of a connection. One possible explanation for these contradictory results comes from neurological evidence, which points to identity recognition being mostly based on holistic processing, while emotion recognition seems to be based on both an explicit, fine-grained process, and an implicit, mostly-holistic one. Our main hypothesis, that would explain the contradictory findings, is that holistic implicit emotion recognition, specifically, would be related to identity recognition, while explicit emotion recognition would be a process separate to identity recognition. To test this hypothesis, we employed an experimental paradigm in which spatial frequencies of visual stimuli are manipulated so that automatic, holistic-based, implicit emotion recognition influences perceived friendliness of unfamiliar faces. We predicted the effect to be related to identity recognition ability, since they both require holistic face processing. After a successful replication study, we employed the paradigm with 140 participants, measuring also identity recognition ability and explicit emotion recognition ability. Results showed that the effect is not moderated by these two variables (p = .807 and .373, respectively), suggesting that the independence of identity and emotion recognition holds even when considering, specifically, implicit emotion recognition.

Development and Validation of the Holistic Cognition Scale

This paper introduces a new scale to measure cognitive cultural differences, drawing on the theory of analytic versus holistic thought. Examining culture from a cognitive perspective is a challenge to traditional values-based approaches. Existing measures based on this framework are methodologically problematic and warrant renewal. This paper presents development and validation studies for a new instrument that measures analytic versus holistic cognitive tendencies at the individual level. The scale assesses four previously established dimensions: attention, causality, contradiction, and change. The present work follows well-established scale development protocols and the results show that the 16-item Holistic Cognition Scale (HCS) is a valid and reliable measure of analytic versus holistic thought. Three new studies with four unique samples (N = 41; 272; 454; and 454) provide evidence to support the content validity, reliability, and factor structure of the new instrument, as well as its convergent, discriminant, and concurrent validity against comparable constructs. Convergent validity is established against measures of compromise, intuition, complexity, and collectivism; predictive validity is established against Hofstede's (1980) five cultural value dimensions; and discriminant validity is established using the average variance extracted from a confirmatory factor analysis. The new HCS is an improvement over previous attempts with a balanced number of forward- and reverse-scored items, superior reliability, less redundancy, and stronger factor loadings.

Facial expression recognition: A meta-analytic review of theoretical models and neuroimaging evidence

Discrimination of facial expressions is an elementary function of the human brain. While the way emotions are represented in the brain has long been debated, common and specific neural representations in recognition of facial expressions are also complicated. To examine brain organizations and asymmetry on discrete and dimensional facial emotions, we conducted an activation likelihood estimation meta-analysis and meta-analytic connectivity modelling on 141 studies with a total of 3138 participants. We found consistent engagement of the amygdala and a common set of brain networks across discrete and dimensional emotions. The left-hemisphere dominance of the amygdala and AI across categories of facial expression, but category-specific lateralization of the vmPFC, suggesting a flexibly asymmetrical neural representations of facial expression recognition. These results converge to characteristic activation and connectivity patterns across discrete and dimensional emotion categories in recognition of facial expressions. Our findings provide the first quantitatively meta-analytic brain network-based evidence supportive of the psychological constructionist hypothesis in facial expression recognition.

Frontotemporal dementia, music perception and social cognition share neurobiological circuits: A meta-analysis

Frontotemporal dementia (FTD) is a neurodegenerative disease that presents with profound changes in social cognition. Music might be a sensitive probe for social cognition abilities, but underlying neurobiological substrates are unclear. We performed a meta-analysis of voxel-based morphometry studies in FTD patients and functional MRI studies for music perception and social cognition tasks in cognitively normal controls to identify robust patterns of atrophy (FTD) or activation (music perception or social cognition). Conjunction analyses were performed to identify overlapping brain regions. In total 303 articles were included: 53 for FTD (n = 1153 patients, 42.5% female; 1337 controls, 53.8% female), 28 for music perception (n = 540, 51.8% female) and 222 for social cognition in controls (n = 5664, 50.2% female). We observed considerable overlap in atrophy patterns associated with FTD, and functional activation associated with music perception and social cognition, mostly encompassing the ventral language network. We further observed overlap across all three modalities in mesolimbic, basal forebrain and striatal regions. The results of our meta-analysis suggest that music perception and social cognition share neurobiological circuits that are affected in FTD. This supports the idea that music might be a sensitive probe for social cognition abilities with implications for diagnosis and monitoring.

Pinpointing the optimal spatial frequency range for automatic neural facial fear processing

Faces convey an assortment of emotional information via low and high spatial frequencies (LSFs and HSFs). However, there is no consensus on the role of particular spatial frequency (SF) information during facial fear processing. Comparison across studies is hampered by the high variability in cut-off values for demarcating the SF spectrum and by differences in task demands. We investigated which SF information is minimally required to rapidly detect briefly presented fearful faces in an implicit and automatic manner, by sweeping through an entire SF range without constraints of predefined cut-offs for LSFs and HSFs. We combined fast periodic visual stimulation with electroencephalography. We presented neutral faces at 6 ​Hz, periodically interleaved every 5th image with a fearful face, allowing us to quantify an objective neural index of fear discrimination at exactly 1.2 ​Hz. We started from a stimulus containing either only very low or very high SFs and gradually increased the SF content by adding higher or lower SF information, respectively, to reach the full SF spectrum over the course of 70 ​s. We found that faces require at least SF information higher than 5.93 cycles per image (cpi) to implicitly differentiate fearful from neutral faces. However, exclusive HSF faces, even in a restricted SF range between 94.82 and 189.63 cpi already carry the critical information to extract the emotional expression of the faces.

Brain Processes While Struggling With Evidence Accumulation During Facial Emotion Recognition: An ERP Study

The human brain is tuned to recognize emotional facial expressions in faces having a natural upright orientation. The relative contributions of featural, configural, and holistic processing to decision-making are as yet poorly understood. This study used a diffusion decision model (DDM) of decision-making to investigate the contribution of early face-sensitive processes to emotion recognition from physiognomic features (the eyes, nose, and mouth) by determining how experimental conditions tapping those processes affect early face-sensitive neuroelectric reflections (P100, N170, and P250) of processes determining evidence accumulation at the behavioral level. We first examined the effects of both stimulus orientation (upright vs. inverted) and stimulus type (photographs vs. sketches) on behavior and neuroelectric components (amplitude and latency). Then, we explored the sources of variance common to the experimental effects on event-related potentials (ERPs) and the DDM parameters. Several results suggest that the N170 indicates core visual processing for emotion recognition decision-making: (a) the additive effect of stimulus inversion and impoverishment on N170 latency; and (b) multivariate analysis suggesting that N170 neuroelectric activity must be increased to counteract the detrimental effects of face inversion on drift rate and of stimulus impoverishment on the stimulus encoding component of non-decision times. Overall, our results show that emotion recognition is still possible even with degraded stimulation, but at a neurocognitive cost, reflecting the extent to which our brain struggles to accumulate sensory evidence of a given emotion. Accordingly, we theorize that: (a) the P100 neural generator would provide a holistic frame of reference to the face percept through categorical encoding; (b) the N170 neural generator would maintain the structural cohesiveness of the subtle configural variations in facial expressions across our experimental manipulations through coordinate encoding of the facial features; and (c) building on the previous configural processing, the neurons generating the P250 would be responsible for a normalization process adapting to the facial features to match the stimulus to internal representations of emotional expressions.

Brain Networks Processing Temporal Information in Dynamic Facial Expressions

This fMRI study examines the role of local and global motion information in facial movements during exposure to novel dynamic face stimuli. We found that synchronous expressions distinctively engaged medial prefrontal areas in the rostral and caudal sectors of anterior cingulate cortex (r/cACC) extending to inferior supplementary motor areas, as well as motor cortex and bilateral superior frontal gyrus (global temporal-spatial processing). Asynchronous expressions in which one part of the face unfolded before the other activated more the right superior temporal sulcus (STS) and inferior frontal gyrus (local temporal-spatial processing). These differences in temporal dynamics had no effect on visual face-responsive areas. Dynamic causal modeling analysis further showed that processing of asynchronous expression features was associated with a differential information flow, centered on STS, which received direct input from occipital cortex and projected to the amygdala. Moreover, STS and amygdala displayed selective interactions with cACC where the integration of both local and global motion cues could take place. These results provide new evidence for a role of local and global temporal dynamics in emotional expressions, extracted in partly separate brain pathways. Importantly, we show that dynamic expressions with synchronous movement cues may distinctively engage brain areas responsible for motor execution of expressions.

Subclinical paranoid beliefs and enhanced neural response during processing of unattractive faces

The perception of faces and consequent social inferences are fundamental for interpersonal communication. While facial expression is important for interindividual communication, constitutional and acquired features are crucial for basic emotions of attraction or repulsion. An emotional bias in face processing has been shown in schizophrenia, but the neurobiological mechanisms are unclear. Studies on the interaction between face processing and the emotional state of healthy individuals may help to elucidate the pathogenesis of the paranoid syndrome in psychosis. This study addressed facial attractiveness and paranoid ideas in a non-clinical population. Using functional magnetic resonance imaging (fMRI), we investigated neural activation patterns of 99 healthy subjects during the passive perception of a dynamic presentation of faces with different attractiveness. We found that the perceived attractiveness of faces was linked to the activity of face processing and limbic regions including the fusiform gyrus, amygdala, and prefrontal areas. Paranoid beliefs interacted with perceived attractiveness in these regions resulting in a higher response range and increased activation after the presentation of unattractive faces. However, no behavioral interactions between reported subjective attractiveness and paranoid beliefs were found. The results showed that increased activation of limbic brain regions is linked to paranoid beliefs. Since similar correlations were found in clinical populations with paranoid syndromes, we suggest a dimension of emotional dysregulation ranging from subclinical paranoid beliefs to paranoid schizophrenia.

Understanding the role of configural processing in face emotion recognition in Parkinson's disease

This investigation examined whether impairment in configural processing could explain deficits in face emotion recognition in people with Parkinson's disease (PD). Stimuli from the Radboud Faces Database were used to compare recognition of four negative emotion expressions by older adults with PD (n = 16) and matched controls (n = 17). Participants were tasked with categorizing emotional expressions from upright and inverted whole faces and facial composites; it is difficult to derive configural information from these two types of stimuli so featural processing should play a larger than usual role in accurate recognition of emotional expressions. We found that the PD group were impaired relative to controls in recognizing anger, disgust and fearful expressions in upright faces. Then, consistent with a configural processing deficit, participants with PD showed no composite effect when attempting to identify facial expressions of anger, disgust and fear. A face inversion effect, however, was observed in the performance of all participants in both the whole faces and facial composites tasks. These findings can be explained in terms of a configural processing deficit if it is assumed that the disruption caused by facial composites was specific to configural processing, whereas inversion reduced performance by making it difficult to derive both featural and configural information from faces.

Atypical holistic processing of facial identity and expression in a case of acquired prosopagnosia

Typical face perception is mediated by holistic processing (i.e., the simultaneous integration of face parts into a whole representation). People with Acquired Prosopagnosia (AP), who have lost the ability to recognise faces after a brain lesion, should thus show atypical holistic coding. Our aim is to use the composite-face effect (CFE) as a measure of holistic processing in ST, a 48-year-old woman with AP but normal recognition of facial expressions of emotions, and matched healthy control participants. Two experiments examining the CFE for identity (Experiment 1) and for expression of emotions (Experiment 2) were conducted. Contrary to controls, in both experiments, ST showed an atypical (i.e., reversed) CFE, thus suggesting altered holistic mechanisms affecting both components of perceptual judgement. Results also suggest that normal facial expression recognition is achievable even with holistic processing difficulties, possibly through compensatory, part-based, mechanisms.

Emotional learning promotes perceptual predictions by remodeling stimulus representation in visual cortex

Emotions exert powerful effects on perception and memory, notably by modulating activity in sensory cortices so as to capture attention. Here, we examine whether emotional significance acquired by a visual stimulus can also change its cortical representation by linking neuronal populations coding for different memorized versions of the same stimulus, a mechanism that would facilitate recognition across different appearances. Using fMRI, we show that after pairing a given face with threat through conditioning, viewing this face activates the representation of another viewpoint of the same person, which itself was never conditioned, leading to robust repetition-priming across viewpoints in the ventral visual stream (including medial fusiform, lateral occipital, and anterior temporal cortex). We also observed a functional-anatomical segregation for coding view-invariant and view-specific identity information. These results indicate emotional signals may induce plasticity of stimulus representations in visual cortex, serving to generate new sensory predictions about different appearances of threat-associated stimuli.

The value of a real face: Differences between affective faces and emojis in neural processing and their social influence on decision-making

Emotional feedback is a crucial part of social interaction, since it may indicate motivations, intentions, and thus, the future behavior of interaction partners. Nowadays, social interaction has been enriched by artificial emotional feedback provided by emojis, which are the means of transporting emotions in mobile messengers. In this study, we examined the influence of emotional feedback by emojis compared to real faces on decision-making and neural processing. We modified the ultimatum game by including proposers represented both by emojis and human faces who reacted specifically toward acceptance or rejection of an offer. We show that proposers who reward acceptance with a smile cause the highest acceptance rates. Interestingly, acceptance rates did not differ between proposers represented by humans compared to emojis. Regarding electrophysiology, emojis evoked more negative N170 and N2 brain potentials compared to human faces both during a mere presentation and as feedback stimuli. Proposers that showed emotional facial expressions evoked larger N170 amplitudes as compared to neutral expressions. Especially the proposers represented by emojis evoked larger P3 amplitudes as feedback stimuli compared to human facial expressions. The comparison of emoji proposers with real-face proposers provides new insight into how relevant social cues influence behavior and its neural underpinnings.

Prefrontal cortex activity triggered by affective faces exposure and its relationship with neuroticism

The emotional processing of affective faces is an essential element of social relationships. Individual differences in personality traits such as neuroticism can influence how we manage these interactions. The objective of this study was to analyze changes in prefrontal cortex (PFC) activity in response to visual exposure to affective faces, and to ascertain whether changes in PFC activity were related to scores in neuroticism, including depression, anxiety, low self-esteem and dependence facets. Fifty-two healthy undergraduate female students participated in the present study. Results showed significant differences depending on face valence in the left and right ventrolateral PFC. We found a reduction in oxygen consumption in reaction to neutral and happy faces, and a small increase in oxygenation in reaction to angry faces both in the left and the right PFC. There were significant positive correlations in the left ventrolateral PFC between oxygenation changes during exposure to neutral and happy faces and the neuroticism factor. Anxiety and depression facets showed positive significant correlations with oxygenation changes for all Time blocks. Notice that participants with high neuroticism scores did not show differences in ventrolateral PFC activity depending on face valence. We suggest that PFC would play a protective role in response to emotional stimuli. The reduced regulatory control of PFC over the amygdala could explain vulnerability to emotional disorders in subjects with high neuroticism.

Confidence of emotion expression recognition recruits brain regions outside the face perception network

Metacognitive beliefs about emotions expressed by others are crucial to social life, yet very little studied. To what extent does our confidence in emotion expression recognition depend on perceptual or other non-perceptual information? We obtained behavioral and magnetic resonance imaging measures while participants judged either the emotion in ambiguous faces or the size of two lines flanking these faces, and then rated their confidence on decision accuracy. Distinct behavioral and neural mechanisms were identified for confidence and perceptual decision in both tasks. Participants overestimated their emotion recognition (ER) accuracy, unlike visual size judgments. Whereas expression discrimination recruited several areas in the face-processing network, confidence for ER uniquely engaged the bilateral retrosplenial/posterior cingulate complex and left parahippocampal gyrus. Further, structural white matter connectivity of the former region predicted metacognitive sensitivity. These results highlight a key role for brain mechanisms integrating perception with contextual mnemonic information in the service of confidence during ER.

Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database

Prior research using static facial stimuli (photographs) has identified diagnostic face regions (i.e., functional for recognition) of emotional expressions. In the current study, we aimed to determine attentional orienting, engagement, and time course of fixation on diagnostic regions. To this end, we assessed the eye movements of observers inspecting dynamic expressions that changed from a neutral to an emotional face. A new stimulus set (KDEF-dyn) was developed, which comprises 240 video-clips of 40 human models portraying six basic emotions (happy, sad, angry, fearful, disgusted, and surprised). For validation purposes, 72 observers categorized the expressions while gaze behavior was measured (probability of first fixation, entry time, gaze duration, and number of fixations). Specific visual scanpath profiles characterized each emotional expression: The eye region was looked at earlier and longer for angry and sad faces; the mouth region, for happy faces; and the nose/cheek region, for disgusted faces; the eye and the mouth regions attracted attention in a more balanced manner for surprise and fear. These profiles reflected enhanced selective attention to expression-specific diagnostic face regions. The KDEF-dyn stimuli and the validation data will be available to the scientific community as a useful tool for research on emotional facial expression processing.

Deep neural network predicts emotional responses of the human brain from functional magnetic resonance imaging

An artificial neural network with multiple hidden layers (known as a deep neural network, or DNN) was employed as a predictive model (DNN_p) for the first time to predict emotional responses using whole-brain functional magnetic resonance imaging (fMRI) data from individual subjects. During fMRI data acquisition, 10 healthy participants listened to 80 International Affective Digital Sound stimuli and rated their own emotions generated by each sound stimulus in terms of the arousal, dominance, and valence dimensions. The whole-brain spatial patterns from a general linear model (i.e., beta-valued maps) for each sound stimulus and the emotional response ratings were used as the input and output for the DNN_P, respectively. Based on a nested five-fold cross-validation scheme, the paired input and output data were divided into training (three-fold), validation (one-fold), and test (one-fold) data. The DNN_P was trained and optimized using the training and validation data and was tested using the test data. The Pearson's correlation coefficients between the rated and predicted emotional responses from our DNN_P model with weight sparsity optimization (mean ± standard error 0.52 ± 0.02 for arousal, 0.51 ± 0.03 for dominance, and 0.51 ± 0.03 for valence, with an input denoising level of 0.3 and a mini-batch size of 1) were significantly greater than those of DNN models with conventional regularization schemes including elastic net regularization (0.15 ± 0.05, 0.15 ± 0.06, and 0.21 ± 0.04 for arousal, dominance, and valence, respectively), those of shallow models including logistic regression (0.11 ± 0.04, 0.10 ± 0.05, and 0.17 ± 0.04 for arousal, dominance, and valence, respectively; average of logistic regression and sparse logistic regression), and those of support vector machine-based predictive models (SVM_ps; 0.12 ± 0.06, 0.06 ± 0.06, and 0.10 ± 0.06 for arousal, dominance, and valence, respectively; average of linear and non-linear SVM_ps). This difference was confirmed to be significant with a Bonferroni-corrected p-value of less than 0.001 from a one-way analysis of variance (ANOVA) and subsequent paired t-test. The weights of the trained DNN_Ps were interpreted and input patterns that maximized or minimized the output of the DNN_Ps (i.e., the emotional responses) were estimated. Based on a binary classification of each emotion category (e.g., high arousal vs. low arousal), the error rates for the DNN_P (31.2% ± 1.3% for arousal, 29.0% ± 1.7% for dominance, and 28.6% ± 3.0% for valence) were significantly lower than those for the linear SVM_P (44.7% ± 2.0%, 50.7% ± 1.7%, and 47.4% ± 1.9% for arousal, dominance, and valence, respectively) and the non-linear SVM_P (48.8% ± 2.3%, 52.2% ± 1.9%, and 46.4% ± 1.3% for arousal, dominance, and valence, respectively), as confirmed by the Bonferroni-corrected p < 0.001 from the one-way ANOVA. Our study demonstrates that the DNN_p model is able to reveal neuronal circuitry associated with human emotional processing - including structures in the limbic and paralimbic areas, which include the amygdala, prefrontal areas, anterior cingulate cortex, insula, and caudate. Our DNN_p model was also able to use activation patterns in these structures to predict and classify emotional responses to stimuli.

Association between Scale-Free Brain Dynamics and Behavioral Performance: Functional MRI Study in Resting State and Face Processing Task

The scale-free dynamics of human brain activity, characterized by an elaborate temporal structure with scale-free properties, can be quantified using the power-law exponent (PLE) as an index. Power laws are well documented in nature in general, particularly in the brain. Some previous fMRI studies have demonstrated a lower PLE during cognitive-task-evoked activity than during resting state activity. However, PLE modulation during cognitive-task-evoked activity and its relationship with an associated behavior remain unclear. In this functional fMRI study in the resting state and face processing + control task, we investigated PLE during both the resting state and task-evoked activities, as well as its relationship with behavior measured using mean reaction time (mRT) during the task. We found that (1) face discrimination-induced BOLD signal changes in the medial prefrontal cortex (mPFC), posterior cingulate cortex (PCC), amygdala, and fusiform face area; (2) PLE significantly decreased during task-evoked activity specifically in mPFC compared with resting state activity; (3) most importantly, in mPFC, mRT significantly negatively correlated with both resting state PLE and the resting-task PLE difference. These results may lead to a better understanding of the associations between task performance parameters (e.g., mRT) and the scale-free dynamics of spontaneous and task-evoked brain activities.

Exogenous attention intensifies perceived emotion expressions

Spatial attention not only enhances early visual processing and improves performance but also alters phenomenology of basic perceptual features. However, in spite of extensive research on attention altering appearance, it is still unknown whether attention also intensifies perceived facial emotional expressions. We investigated the effect of exogenous attention on two categories of emotions, one positive (happy) and one negative (sad) in separate sessions. Exogenous attention was manipulated using peripheral cues followed by two faces varying in emotional intensity that were presented on either side of fixation. Participants were asked to report the location of the emotional face displaying higher intensity of emotion. At short cue-to-target interval [CTI, Experiment 1 (60 ms)], participants reported the cued emotional face as more intense in expression compared with the uncued face. However, at longer CTI [Experiment 2 (500 ms)], this effect was absent. Results show that exogenous attention enhances appearance of higher level features, such as emotional intensity, irrespective of valence. Further, two experiments investigated the mediating role of facial contrast as a possible underlying mechanism for the observed effect. Although the results show that higher contrast faces are judged as more in emotional intensity, spatial attention effects seem to be dependent on task instructions. Possible mechanisms underlying the attentional effects on emotion intensity are discussed.