Early brain responses to affective faces: A simultaneous EEG-fMRI study

Is processing superiority a universal trait for all threats? Divergent impacts of fearful, angry, and disgusted faces on attentional capture

Fearful, angry, and disgusted facial expressions are evolutionarily salient and convey different types of threat signals. However, it remains unclear whether these three expressions impact sensory perception and attention in the same way. The present ERP study investigated the temporal dynamics underlying the processing of different types of threatening faces and the impact of attentional resources employed during a perceptual load task. Participants were asked to judge the length of bars superimposed over faces presented in the center of the screen. A mass univariate statistical approach was used to analyze the EEG data. Behaviorally, task accuracy was significantly reduced following exposure to fearful faces relative to neutral distractors, independent of perceptual load. The ERP results revealed that the P1 amplitude over the right hemisphere was found to be enhanced for fearful relative to disgusted faces, reflecting the rapid and coarse detection of fearful cues. The N170 responses elicited by fearful, angry, and disgusted faces were larger than those elicited by neutral faces, suggesting the largely automatic and preferential processing of threats. Furthermore, the early posterior negativity (EPN) component yielded increased responses to fearful and angry faces, indicating prioritized attention to stimuli representing acute threats. Additionally, perceptual load exerted a pronounced influence on the EPN and late positive potential (LPP), with larger responses observed in the low perceptual load condition, indicating goal-directed cognitive processing. Overall, the early sensory processing of fearful, angry, and disgusted faces is characterized by differential sensitivity in capturing attention automatically, despite the importance of these facial signals for survival. Fearful faces produce a strong interference effect and are processed with higher priority than angry and disgusted ones.

Prioritized neural processing of social threats during perceptual decision-making

Emotional signals, notably those signaling threat, benefit from prioritized processing in the human brain. Yet, it remains unclear whether perceptual decisions about the emotional, threat-related aspects of stimuli involve specific or similar neural computations compared to decisions about their non-threatening/non-emotional components. We developed a novel behavioral paradigm in which participants performed two different detection tasks (emotion vs. color) on the same, two-dimensional visual stimuli. First, electroencephalographic (EEG) activity in a cluster of central electrodes reflected the amount of perceptual evidence around 100 ms following stimulus onset, when the decision concerned emotion, not color. Second, participants' choice could be predicted earlier for emotion (240 ms) than for color (380 ms) by the mu (10 Hz) rhythm, which reflects motor preparation. Taken together, these findings indicate that perceptual decisions about threat-signaling dimensions of facial displays are associated with prioritized neural coding in action-related brain regions, supporting the motivational value of socially relevant signals.

Amygdala and cortical gamma-band responses to emotional faces are modulated by attention to valence

The amygdala might support an attentional bias for emotional faces. However, whether and how selective attention toward a specific valence modulates this bias is not fully understood. Likewise, it is unclear whether amygdala and cortical signals respond to emotion and attention in a similar way. We recorded gamma-band activity (GBA, > 30 Hz) intracranially in the amygdalae of 11 patients with epilepsy and collected scalp recordings from 19 healthy participants. We presented angry, neutral, and happy faces randomly, and we denoted one valence as the target. Participants detected happy targets most quickly and accurately. In the amygdala, during attention to negative faces, low gamma-band activity (LGBA, < 90 Hz) increased for angry compared with happy faces from 160 ms. From 220 ms onward, amygdala high gamma-band activity (HGBA, > 90 Hz) was higher for angry and neutral faces than for happy ones. Monitoring neutral faces increased amygdala HGBA for emotions compared with neutral faces from 40 ms. Expressions were not differentiated in GBA while monitoring positive faces. On the scalp, only threat monitoring resulted in expression differentiation. Here, posterior LGBA was increased selectively for angry targets from 60 ms. The data show that GBA differentiation of emotional expressions is modulated by attention to valence: Top-down-controlled threat vigilance coordinates widespread GBA in favor of angry faces. Stimulus-driven emotion differentiation in amygdala GBA occurs during a neutral attentional focus. These findings align with a multi-pathway model of emotion processing and specify the role of GBA in this process.

How and when social evaluative feedback is processed in the brain: A systematic review on ERP studies

Social evaluative feedback informs the receiver of the other's views, which may contain judgments of personality-related traits and/or the level of likability. Such kinds of social evaluative feedback are of particular importance to humans. Event-related potentials (ERPs) can directly measure where in the processing stream feedback valence, expectancy, or contextual relevance modulate information processing. This review provides an overview and systematization of studies and early, mid-latency, and late ERP effects. Early effects were inconsistently reported for all factors. Feedback valence effects are more consistently reported for specific mid-latency ERPs (Reward Positivity, RewP, and Early Posterior Negativity, EPN) and late positivities (P3 and Late Positive Potential, LPP). Unexpected feedback consistently increased the Feedback Related Negativity (FRN) and, less consistently, decreased P3 amplitudes. Contextual relevance of the sender (e.g., human vs computer sender) or self-relatedness increased mid-latency to late ERPs. Interactions between valence and other factors were less often found, arising during mid-latency stages, where most consistent interactions showed larger EPN and P3 amplitude differences for valent feedback in a more relevant context. The ERP findings highlight that social evaluative feedback is consistently differentiated during mid-latency processing stages. The review discusses the relevance of findings, possible shortcomings of different experimental designs, and open questions. Furthermore, we suggest concrete venues for future research.

Differential role of fusiform gyrus coupling in depressive and anxiety symptoms during emotion perception

Anxiety and depression co-occur; the neural substrates of shared and unique components of these symptoms are not understood. Given emotional alterations in internalizing disorders, we hypothesized that function of regions associated with emotion processing/regulation, including the anterior cingulate cortex (ACC), amygdala and fusiform gyrus (FG), would differentiate these symptoms. Forty-three adults with depression completed an emotional functional magnetic resonance imaging task and the Hamilton Depression and Anxiety Scales. We transformed these scales to examine two orthogonal components, one representing internalizing symptom severity and the other the type of internalizing symptoms (anxiety vs depression). We extracted blood oxygen level dependent signal from FG subregions, ACC, and amygdala and performed generalized psychophysiological interaction analyses to assess relationships between symptoms and brain function. Type of internalizing symptoms was associated with FG3-FG1 coupling (F = 8.14, P = 0.007). More coupling was associated with a higher concentration of depression, demonstrating that intra-fusiform coupling is differentially associated with internalizing symptom type (anxiety vs depression). We found an interaction between task condition and internalizing symptoms and dorsal (F = 4.51, P = 0.014) and rostral ACC activity (F = 4.27, P = 0.012). Post hoc comparisons revealed that less activity was associated with greater symptom severity during emotional regulation. Functional coupling differences during emotional processing are associated with depressive relative to anxiety symptoms and internalizing symptom severity. These findings could inform future treatments for depression.

Temporal characteristics of emotional conflict processing: the modulation role of attachment styles

Theoretical account of attachment proposed that individual differences in adult attachment styles play a key role in adjusting balance between affective evaluation and cognitive control. Yet, little is known about the temporal characteristics of emotional conflict processing modulated by attachment styles. Accordingly, the present study used event-related potentials (ERP) and multivariate pattern analysis (MVPA) combined with an emotional face-word Stroop task to investigate the temporal dynamics of attachment-related cognitive-affective patterns in emotional conflict processing. The ERP results demonstrated multiple-process of emotional conflict modulated by attachment styles. In early sensory processing, positive faces captured avoidant attachment individuals' attention as reflected in greater P1, while the same situation led to greater N170 in secure and anxious individuals. Crucially, impairment in conflict-monitoring function was found in anxious individuals as reflected by the absence of interference effect on N450, leading to impaired ability of inhibitory control as indicated by decreased slow potential. In contrast, avoidant individuals showed greater slow potential for inhibiting emotional interference. Furthermore, MVPA revealed that the corresponding time window for conflict monitoring was found for emotional distractors decoding rather than congruency decoding in the anxious attachment group. Convergent results from ERPs and MVPA indicated that the deficits in emotional conflict monitoring and resolution among anxious individuals might be due to the excessive approach to emotional distractors, as they habitually use emotional evaluation rather than cognitive control. In summary, the present study provides electrophysiological evidence that attachment styles modulated emotional conflict processing, which highlights the contribution of attachment to social information processing.

Using multi-modal neuroimaging to characterise social brain specialisation in infants

The specialised regional functionality of the mature human cortex partly emerges through experience-dependent specialisation during early development. Our existing understanding of functional specialisation in the infant brain is based on evidence from unitary imaging modalities and has thus focused on isolated estimates of spatial or temporal selectivity of neural or haemodynamic activation, giving an incomplete picture. We speculate that functional specialisation will be underpinned by better coordinated haemodynamic and metabolic changes in a broadly orchestrated physiological response. To enable researchers to track this process through development, we develop new tools that allow the simultaneous measurement of coordinated neural activity (EEG), metabolic rate, and oxygenated blood supply (broadband near-infrared spectroscopy) in the awake infant. In 4- to 7-month-old infants, we use these new tools to show that social processing is accompanied by spatially and temporally specific increases in coupled activation in the temporal-parietal junction, a core hub region of the adult social brain. During non-social processing, coupled activation decreased in the same region, indicating specificity to social processing. Coupling was strongest with high-frequency brain activity (beta and gamma), consistent with the greater energetic requirements and more localised action of high-frequency brain activity. The development of simultaneous multimodal neural measures will enable future researchers to open new vistas in understanding functional specialisation of the brain.

Beyond facial expressions: A systematic review on effects of emotional relevance of faces on the N170

The N170 is the most prominent electrophysiological signature of face processing. While facial expressions reliably modulate the N170, there is considerable variance in N170 modulations by other sources of emotional relevance. Therefore, we systematically review and discuss this research area using different methods to manipulate the emotional relevance of inherently neutral faces. These methods were categorized into (1) existing pre-experimental affective person knowledge (e.g., negative attitudes towards outgroup faces), (2) experimentally instructed affective person knowledge (e.g., negative person information), (3) contingency-based affective learning (e.g., fear-conditioning), or (4) the immediate affective context (e.g., emotional information directly preceding the face presentation). For all categories except the immediate affective context category, the majority of studies reported significantly increased N170 amplitudes depending on the emotional relevance of faces. Furthermore, the potentiated N170 was observed across different attention conditions, supporting the role of the emotional relevance of faces on the early prioritized processing of configural facial information, regardless of low-level differences. However, we identified several open research questions and suggest venues for further research.

The influence of selective attention to specific emotions on the processing of faces as revealed by event-related brain potentials

Event-related potential studies using affective words have indicated that selective attention to valence can increase affective discrimination at early perceptual stages. This effect most likely relies on neural associations between perceptual features of a stimulus and its affective value. Similar to words, emotional expressions in human faces are linked to specific visual elements. Therefore, selectively attending to a given emotion should allow for the preactivation of neural networks coding for the emotion and associated first-order visual elements, leading to enhanced early processing of faces expressing the attended emotion. To investigate this, we employed an expression detection task (N = 65). Fearful, happy, and neutral faces were randomly presented in three blocks while participants were instructed to respond only to one predefined target level of expression in each block. Reaction times were the fastest for happy target faces, which was accompanied by an increased occipital P1 for happy compared with fearful faces. The N170 yielded an arousal effect (emotional > neutral) while both components were not modulated by target status. In contrast, the early posterior negativity (EPN) arousal effect tended to be larger for target compared with nontarget faces. The late positive potential (LPP) revealed large effects of status and expression as well as an interaction driven by an increased LPP specifically for nontarget fearful faces. These findings tentatively indicate that selective attention to facial affect may enhance early emotional processing (EPN) even though further research is needed. Moreover, late controlled processing of facial emotions appears to involve a negativity bias.

Reward History Modulates the Processing of Task-Irrelevant Emotional Faces in a Demanding Task

The aim of the current study was to examine how reward-associated emotional facial distractors could capture attentional resources in a demanding visual task using event-related potentials (ERPs). In the learning phase, a high- or low-reward probability was paired with angry, happy, or neutral faces. Then, in the test phase, participants performed a face-irrelevant task with no reward at stake, in which they needed to discriminate the length of two lines presented in the center of the screen while faces that were taken from the learning phase were used as distractors presented in the periphery. The behavioral results revealed no effect of distractor emotional valence since the emotional information was task-irrelevant. The ERP results in the test phase revealed a significant main effect of distractor emotional valence for the parieto-occipital P200 (170-230 ms); the mean amplitudes in both the angry- and happy-face conditions were more positive than the neutral-face condition. Moreover, we found that the high-reward association enhanced both the N170 (140-180 ms) and EPN (260-330 ms) relative to the low-reward association condition. Finally, the N2pc (270-320 ms) also exhibited enhanced neural activity in the high-reward condition compared to the low-reward condition. The absence of emotional effects indicated that task-irrelevant emotional facial stimuli did not impact behavioral or neural responses in this highly demanding task. However, reward-associated information was processed when attention was directed elsewhere, suggesting that the processing of reward-associated information worked more in an automatic way, irrespective of the top-down task demand.

Early neural potentiation to centrally and peripherally presented fear-conditioned faces

For humans, it is vitally important to rapidly detect and process threatening signals regardless of whether stimuli occur at fixation or in the periphery. However, it is unknown whether eccentricity affects early neuronal electrophysiological responses to fear-conditioned stimuli. We examined early event-related potentials (ERPs) of the electroencephalogram (EEG) to fear-conditioned faces to address this question. Participants (N = 80) were presented with faces, either paired with an aversive (CS+) or neutral sound (CS-), at central or peripheral positions. We ensured constant central fixation using online eye-tracking but directed attention to either centrally or peripherally presented faces. Manipulation checks showed successful fear-conditioning (i.e., on average lower ratings in valence and higher ratings in arousal and perceived threat) and successful shifts of visuospatial attention indexed by high task performance and pre-stimulus alpha lateralization of the EEG spectra. We observed a generally increased P1 to fear-conditioned faces regardless of presentation location. An N170 difference between fear-conditioned and neutral stimuli was found but was restricted to the central location and depended on the effectivity of fear-conditioning. A similar effect was observed for the early posterior negativity (EPN). Trait anxiety was not related to differential ERP responses to CS+ versus CS- faces for any ERP component. These findings suggest that the P1 indexes early responses to centrally and peripherally presented fear-conditioned faces. Subsequent stages are modulated by the spatial location of the stimuli. This suggests different stages of neural processing of fear-conditioned faces depending on their spatial location. Finally, our results question the hypothesis that trait anxiety in healthy participants is related to altered visual processing of fear-conditioned faces.

The effect of sad mood on early sensory event-related potentials to task-irrelevant faces

It has been shown that the perceiver's mood affects the perception of emotional faces, but it is not known how mood affects preattentive brain responses to emotional facial expressions. To examine the question, we experimentally induced sad and neutral mood in healthy adults before presenting them with task-irrelevant pictures of faces while an electroencephalography was recorded. Sad, happy, and neutral faces were presented to the participants in an ignore oddball condition. Differential responses (emotional - neutral) for the P1, N170, and P2 amplitudes were extracted and compared between neutral and sad mood conditions. Emotional facial expressions modulated all the components, and an interaction effect of expression by mood was found for P1: an emotional modulation to happy faces, which was found in neutral mood condition, disappeared in sad mood condition. For N170 and P2, we found larger response amplitudes for both emotional faces, regardless of the mood. The results add to the previous behavioral findings showing that mood already affects low-level cortical feature encoding of task-irrelevant faces.

The neural correlates of context driven changes in the emotional response: An fMRI study

Emotional flexibility reflects the ability to adjust the emotional response to the changing environmental context. To understand how context can trigger a change in emotional response, i.e., how it can upregulate the initial emotional response or trigger a shift in the valence of emotional response, we used a task consisting of picture pairs during functional magnetic resonance imaging sessions. In each pair, the first picture was a smaller detail (a decontextualized photograph depicting emotions using primarily facial and postural expressions) from the second (contextualized) picture, and the neural response to a decontextualized picture was compared with the same picture in a context. Thirty-one healthy participants (18 females; mean age: 24.44 ± 3.4) were involved in the study. In general, context (vs. pictures without context) increased activation in areas involved in facial emotional processing (e.g., middle temporal gyrus, fusiform gyrus, and temporal pole) and affective mentalizing (e.g., precuneus, temporoparietal junction). After excluding the general effect of context by using an exclusive mask with activation to context vs. no-context, the automatic shift from positive to negative valence induced by the context was associated with increased activation in the thalamus, caudate, medial frontal gyrus and lateral orbitofrontal cortex. When the meaning changed from negative to positive, it resulted in a less widespread activation pattern, mainly in the precuneus, middle temporal gyrus, and occipital lobe. Providing context cues to facial information recruited brain areas that induced changes in the emotional responses and interpretation of the emotional situations automatically to support emotional flexibility.

Representational structure of fMRI/EEG responses to dynamic facial expressions

Face perception provides an excellent example of how the brain processes nuanced visual differences and transforms them into behaviourally useful representations of identities and emotional expressions. While a body of literature has looked into the spatial and temporal neural processing of facial expressions, few studies have used a dimensionally varying set of stimuli containing subtle perceptual changes. In the current study, we used 48 short videos varying dimensionally in their intensity and category (happy, angry, surprised) of expression. We measured both fMRI and EEG responses to these video clips and compared the neural response patterns to the predictions of models based on image features and models derived from behavioural ratings of the stimuli. In fMRI, the inferior frontal gyrus face area (IFG-FA) carried information related only to the intensity of the expression, independent of image-based models. The superior temporal sulcus (STS), inferior temporal (IT) and lateral occipital (LO) areas contained information about both expression category and intensity. In the EEG, the coding of expression category and low-level image features were most pronounced at around 400 ms. The expression intensity model did not, however, correlate significantly at any EEG timepoint. Our results show a specific role for IFG-FA in the coding of expressions and suggest that it contains image and category invariant representations of expression intensity.

Emotion and attention in face processing: Complementary evidence from surface event-related potentials and intracranial amygdala recordings

Face processing is biased by emotional and voluntarily directed attention, both of which modulate processing in distributed cortical areas. The amygdala is assumed to contribute to an attentional bias for emotional faces, although its interaction with directed attention awaits further clarification. Here, we studied the interaction of emotion and attention during face processing via scalp EEG potentials of healthy participants and intracranial EEG (iEEG) recordings of the right amygdala in one patient. Three randomized blocks consisting of angry, neutral, and happy facial expressions were presented, and one expression was denoted as the target category in each block. Happy targets were detected fastest and most accurately both in the group study and by the iEEG patient. Occipital scalp potentials revealed emotion differentiation for happy faces in the early posterior negativity (EPN) around 300 ms after stimulus onset regardless of the target condition. A similar response to happy faces occurred in the amygdala only for happy targets. On the scalp, a late positive potential (LPP, around 600 ms) enhancement for targets occurred for all target conditions alike. A simultaneous late signal in the amygdala was largest for emotional targets. No late signal enhancements were found for neutral targets in the amygdala. Cortical modulations, by contrast, showed both attention-independent effects of emotion and emotion-independent effects of attention. These results demonstrate an attention-dependence of amygdala activity during the processing of facial expressions and partly independent cortical mechanisms.

Effects of task load, spatial attention, and trait anxiety on neuronal responses to fearful and neutral faces

There is an ongoing debate on how different components of the event-related potential (ERP) to threat-related facial expressions are modulated by attentional conditions and interindividual differences in trait anxiety. In the current study (N = 80), we examined ERPs to centrally presented, task-irrelevant fearful and neutral faces, while participants had to solve a face-unrelated visual task, which differed in difficulty and spatial position. Critically, we used a fixation-controlled experimental design and ensured the spatial attention manipulation by spectral analysis of the EEG. Besides the factors emotion, spatial attention, and perceptual load, we also investigated correlations between trait anxiety and ERPs. While P1 emotion effects were insignificant, the N170 was increased to fearful faces regardless of load and spatial attention conditions. During the EPN time window, a significantly increased negativity for fearful faces was observed only during low load and spatial attention to the face. We found no significant relationship between ERPs and trait anxiety, questioning the hypothesis of a general hypersensitivity toward fearful expressions in anxious individuals. These results show a high resistance of the N170 amplitude increase for fearful faces to spatial attention and task load manipulations. By contrast, the EPN modulation by fearful faces index a resource-dependent stage of the ERP, requiring both spatial attention at the location of faces and low load of the face-irrelevant task.

Neural signatures of individual variability in context-dependent perception of ambiguous facial expression

How do we incorporate contextual information to infer others' emotional state? Here we employed a naturalistic context-dependent facial expression estimation task where participants estimated pleasantness levels of others' ambiguous expression faces when sniffing different contextual cues (e.g., urine, fish, water, and rose). Based on their pleasantness rating data, we placed participants on a context-dependency continuum and mapped the individual variability in the context-dependency onto the neural representation using a representational similarity analysis. We found that the individual variability in the context-dependency of facial expression estimation correlated with the activity level of the pregenual anterior cingulate cortex (pgACC) and the amygdala and was also decoded by the neural representation of the ventral anterior insula (vAI). A dynamic causal modeling revealed that those with higher context-dependency exhibited a greater degree of the modulation from vAI to the pgACC. These findings provide novel insights into the neural circuitry associated with the individual variability in context-dependent facial expression estimation and the first empirical evidence for individual variability in the predictive accounts of affective states.

Enhanced early ERP responses to looming angry faces

Although the brain is known to process threatening emotional stimuli and looming motion rapidly, little is known about how the emotion and motion interact. To address this question, two experiments were carried out which presented angry and neutral emotional faces on a depth-cued background that induced the perception of distance, or a non-cued background. Furthermore, faces either expanded or contracted in size such that they appeared to approach or recede from the viewer. EEG/ERP measures were used to identify the time course of brain activity for these looming and receding, angry and neutral emotional faces. The results of both experiments revealed that the P1 was enhanced by looming angry faces on the depth-cued background, compared to neutral approaching faces, as well as all receding faces, indicating an early interaction of emotion and motion within 100 ms of presentation. Angry expressions were also found to enhance the N170 regardless of movement. These findings suggest that processing of threat and looming motion interact at the very early stages of visual processing. Furthermore, as the modulating effect of looming motion on angry expressions only arose on the depth-cued background, the findings highlight the importance of approaching movements rather than sole increases in the retinal size of the stimuli.

Unpredictable fearful stimuli disrupt timing activities: Evidence from event-related potentials

The present study investigated the effect of an imminent fearful stimulus on an ongoing temporal task. Participants judged the duration of a blank temporal interval followed by a fearful or a neutral image. Results showed an underestimation of the duration in the fearful condition relative to the neutral condition, but only when the occurrence of the fearful image was difficult to predict. ERPs results for the blank temporal interval found no effect of the fearful stimulus on the contingent negative variation (CNV) amplitude in the clock stage. However, after the image onset, there was a larger P1 for the fearful relative to the neutral condition. Although this effect was indistinguishable regardless of whether the fearful event could be easily predicted, a late positive potential (LPP) component displayed larger amplitude only for unpredictable fearful stimuli. The time-frequency results showed enhanced delta-theta power (0.5-7.5 Hz) for the unpredictable fearful stimuli in the late stage. Importantly, the enhanced delta-theta rhythm correlated negatively with the duration judgments. Together, these results suggest that an unpredictable fearful event might divert more attention away from the counting process in the working memory stage, resulting in missing ticks and temporal underestimation.

Localizing Epileptic Foci Using Simultaneous EEG-fMRI Recording: Template Component Cross-Correlation

Conventional EEG-fMRI methods have been proven to be of limited use in the sense that they cannot reveal the information existing in between the spikes. To resolve this issue, the current study obtains the epileptic components time series detected on EEG and uses them to fit the Generalized Linear Model (GLM), as a substitution for classical regressors. This approach allows for a more precise localization, and equally importantly, the prediction of the future behavior of the epileptic generators. The proposed method approaches the localization process in the component domain, rather than the electrode domain (EEG), and localizes the generators through investigating the spatial correlation between the candidate components and the spike template, as well as the medical records of the patient. To evaluate the contribution of EEG-fMRI and concordance between fMRI and EEG, this method was applied on the data of 30 patients with refractory epilepsy. The results demonstrated the significant numbers of 29 and 24 for concordance and contribution, respectively, which mark improvement as compared to the existing literature. This study also shows that while conventional methods often fail to properly localize the epileptogenic zones in deep brain structures, the proposed method can be of particular use. For further evaluation, the concordance level between IED-related BOLD clusters and Seizure Onset Zone (SOZ) has been quantitatively investigated by measuring the distance between IED/SOZ locations and the BOLD clusters in all patients. The results showed the superiority of the proposed method in delineating the spike-generating network compared to conventional EEG-fMRI approaches. In all, the proposed method goes beyond the conventional methods by breaking the dependency on spikes and using the outside-the-scanner spike templates and the selected components, achieving an accuracy of 97%. Doing so, this method contributes to improving the yield of EEG-fMRI and creates a more realistic perception of the neural behavior of epileptic generators which is almost without precedent in the literature.

The rapid and automatic categorization of facial expression changes in highly variable natural images

Emotional expressions are quickly and automatically read from human faces under natural viewing conditions. Yet, categorization of facial expressions is typically measured in experimental contexts with homogenous sets of face stimuli. Here we evaluated how the 6 basic facial emotions (Fear, Disgust, Happiness, Anger, Surprise or Sadness) can be rapidly and automatically categorized with faces varying in head orientation, lighting condition, identity, gender, age, ethnic origin and background context. High-density electroencephalography was recorded in 17 participants viewing 50 s sequences with natural variable images of neutral-expression faces alternating at a 6 Hz rate. Every five stimuli (1.2 Hz), variable natural images of one of the six basic expressions were presented. Despite the wide physical variability across images, a significant F/5 = 1.2 Hz response and its harmonics (e.g., 2F/5 = 2.4 Hz, etc.) was observed for all expression changes at the group-level and in every individual participant. Facial categorization responses were found mainly over occipito-temporal sites, with distinct hemispheric lateralization and cortical topographies according to the different expressions. Specifically, a stronger response was found to Sadness categorization, especially over the left hemisphere, as compared to Fear and Happiness, together with a right hemispheric dominance for categorization of Fearful faces. Importantly, these differences were specific to upright faces, ruling out the contribution of low-level visual cues. Overall, these observations point to robust rapid and automatic facial expression categorization processes in the human brain.

Same salience, different consequences: Disturbed inter-network connectivity during a social oddball paradigm in major depressive disorder

•

No significant difference in BOLD responses in the SN between controls and patients.

•

Anhedonic symptomatology relates to diminished BOLD signals in the left amygdala.

•

Enhanced functional connectivity from the SN to the DMN in depression.

•

Reduced functional connectivity from the SN to the VAN in depression.

•

Altered processing in depression is coded in disturbed inter-network connectivity.

Background

So far findings on emotional face processing among depressed individuals reveal an inconsistent image, with only some studies supporting a mood-congruent bias in salience processing. Thereby, many results are based on the processing of sad emotions and mostly focused on resting-state connectivity analysis. The present study aimed to target this misbalance by implementing a social oddball paradigm, with a special focus on the amygdala, the ACC, the insula and subdivisions of insula and ACC.

Methods

Twenty-seven depressed patients and twenty-seven non-depressed controls took part in a fMRI event-related social oddball paradigm based on smiling facial expressions as target stimuli embedded in a stream of neutral facial expressions. FMRI activation and functional connectivity analysis were calculated for the pre-defined ROIs of the salience network (SN), with a special focus on twelve insular subdivisions and six ACC subdivisions.

Results

For both groups the social oddball paradigm triggered similar BOLD responses within the pre-defined ROIs, while the quality of functional connectivity showed pronounced alterations from the salience network to the ventral attention- and default mode network (DMN).

Conclusion

On a first level of target detection, smiling faces are equally processed and trigger similar bold responses in structures of the salience network. On a second level of inter-network communication the brain of depressed participants tends to be pre-formed for self-referential processing and rumination instead of fast goal directed behavior and socio-emotional cognitive processing.

Representation of the brain network by electroencephalograms during facial expressions

BACKGROUND

Facial expressions, such as smiling and anger, cause many physical and psychological effects in the body, known as 'embodied emotions' or 'facial feedback theory.' In the clinical application of this theory in certain diseases, such as autism and depression, treatments such as forcing patients to smile have been used. However, the neural mechanisms underlying the representation of facial expressions remain unclear.

NEW METHOD

We proposed a method to construct brain networks based on the time course of the synchronization likelihood and determine the effects of various facial expressions on the situation using visual stimulus of faces. This method was applied to analyze electroencephalographic (EEG) data recorded during the recognition and representation of various positive and negative facial expressions. The brain networks were constructed based on the EEG data recorded in 11 healthy participants.

RESULTS

Channel sets from brain networks during unsymmetrical smiling expressions (i.e., only the right or left side) were highly linearly symmetrical. Channel sets from brain networks during negative facial expressions (i.e., anger and sadness) and symmetrical smiling expressions (i.e., smiling with an opened or closed mouth) were similar.

COMPARISON WITH EXISTING METHODS

While we obtained brain networks based on time course EEG correlations throughout the experiment, existing methods can analyze EEG data only at a certain time point.

CONCLUSIONS

The comparisons of different facial expressions could be used to identify the side of the facial muscles used while smiling and to determine how similar brain networks are induced by positive and negative facial expressions.

Face and emotional expression processing under continuous perceptual load tasks: An ERP study

High perceptual load is thought to impair already the early stages of visual processing of task-irrelevant visual stimuli. However, recent studies showed no effects of perceptual load on early ERPs in response to task-irrelevant emotional faces. In this preregistered EEG study (N = 40), we investigated the effects of continuous perceptual load on ERPs to fearful and neutral task-irrelevant faces and their phase-scrambled versions. Perceptual load did not modulate face or emotion effects for the P1 or N170. In contrast, larger face-scramble and fearful-neutral differentiation were found during low as compared to high load for the Early Posterior Negativity (EPN). Further, face-independent P1, but face-dependent N170 emotional modulations were observed. Taken together, our findings show that P1 and N170 face and emotional modulations are highly resistant to load manipulations, indicating a high degree of automaticity during this processing stage, whereas the EPN might represent a bottleneck in visual information processing.

Effects of low-level visual information and perceptual load on P1 and N170 responses to emotional expressions

Emotional facial expressions lead to modulations of early event-related potentials (ERPs). However, it has so far remained unclear how far these modulations represent face-specific effects rather than differences in low-level visual features, and to which extent they depend on available processing resources. To examine these questions, we conducted two preregistered independent experiments (N = 40 in each experiment) using different variants of a novel task that manipulates peripheral perceptual load across levels but keeps overall visual stimulation constant. At the display center, we presented task-irrelevant angry, neutral, and happy faces and their Fourier phase-scrambled versions, which preserved low-level visual features. The results of both studies showed load-independent P1 and N170 emotional expression effects. Importantly, by using Bayesian analyses we could confirm that these facial expression effects were face-independent for the P1 but not for the N170 component. We conclude that firstly, ERP modulations during the P1 interval strongly depend on low-level visual information, while the N170 modulation requires the processing of figural facial expression features. Secondly, both P1 and N170 modulations appear to be immune to a large range of variations in perceptual load.

Attentional conditions differentially affect early, intermediate and late neural responses to fearful and neutral faces

The processing of fearful facial expressions is prioritized by the human brain. This priority is maintained across various information processing stages as evident in early, intermediate and late components of event-related potentials (ERPs). However, emotional modulations are inconsistently reported for these different processing stages. In this pre-registered study, we investigated how feature-based attention differentially affects ERPs to fearful and neutral faces in 40 participants. The tasks required the participants to discriminate either the orientation of lines overlaid onto the face, the sex of the face or the face's emotional expression, increasing attention to emotion-related features. We found main effects of emotion for the N170, early posterior negativity (EPN) and late positive potential (LPP). While N170 emotional modulations were task-independent, interactions of emotion and task were observed for the EPN and LPP. While EPN emotion effects were found in the sex and emotion tasks, the LPP emotion effect was mainly driven by the emotion task. This study shows that early responses to fearful faces are task-independent (N170) and likely based on low-level and configural information while during later processing stages, attention to the face (EPN) or-more specifically-to the face's emotional expression (LPP) is crucial for reliable amplified processing of emotional faces.

Brain Processes While Struggling With Evidence Accumulation During Facial Emotion Recognition: An ERP Study

The human brain is tuned to recognize emotional facial expressions in faces having a natural upright orientation. The relative contributions of featural, configural, and holistic processing to decision-making are as yet poorly understood. This study used a diffusion decision model (DDM) of decision-making to investigate the contribution of early face-sensitive processes to emotion recognition from physiognomic features (the eyes, nose, and mouth) by determining how experimental conditions tapping those processes affect early face-sensitive neuroelectric reflections (P100, N170, and P250) of processes determining evidence accumulation at the behavioral level. We first examined the effects of both stimulus orientation (upright vs. inverted) and stimulus type (photographs vs. sketches) on behavior and neuroelectric components (amplitude and latency). Then, we explored the sources of variance common to the experimental effects on event-related potentials (ERPs) and the DDM parameters. Several results suggest that the N170 indicates core visual processing for emotion recognition decision-making: (a) the additive effect of stimulus inversion and impoverishment on N170 latency; and (b) multivariate analysis suggesting that N170 neuroelectric activity must be increased to counteract the detrimental effects of face inversion on drift rate and of stimulus impoverishment on the stimulus encoding component of non-decision times. Overall, our results show that emotion recognition is still possible even with degraded stimulation, but at a neurocognitive cost, reflecting the extent to which our brain struggles to accumulate sensory evidence of a given emotion. Accordingly, we theorize that: (a) the P100 neural generator would provide a holistic frame of reference to the face percept through categorical encoding; (b) the N170 neural generator would maintain the structural cohesiveness of the subtle configural variations in facial expressions across our experimental manipulations through coordinate encoding of the facial features; and (c) building on the previous configural processing, the neurons generating the P250 would be responsible for a normalization process adapting to the facial features to match the stimulus to internal representations of emotional expressions.

Vocal threat enhances visual perception as a function of attention and sex

This pre-registered event-related potential study explored how vocal emotions shape visual perception as a function of attention and listener sex. Visual task displays occurred in silence or with a neutral or an angry voice. Voices were task-irrelevant in a single-task block, but had to be categorized by speaker sex in a dual-task block. In the single task, angry voices increased the occipital N2 component relative to neutral voices in women, but not men. In the dual task, angry voices relative to neutral voices increased occipital N1 and N2 components, as well as accuracy, in women and marginally decreased accuracy in men. Thus, in women, vocal anger produced a strong, multifaceted visual enhancement comprising attention-dependent and attention-independent processes, whereas in men, it produced a small, behavior-focused visual processing impairment that was strictly attention-dependent. In sum, these data indicate that attention and listener sex critically modulate whether and how vocal emotions shape visual perception.

Modulation of face- and emotion-selective ERPs by the three most common types of face image manipulations

In neuroscientific studies, the naturalness of face presentation differs; a third of published studies makes use of close-up full coloured faces, a third uses close-up grey-scaled faces and another third employs cutout grey-scaled faces. Whether and how these methodological choices affect emotion-sensitive components of the event-related brain potentials (ERPs) is yet unclear. Therefore, this pre-registered study examined ERP modulations to close-up full-coloured and grey-scaled faces as well as cutout fearful and neutral facial expressions, while attention was directed to no-face oddballs. Results revealed no interaction of face naturalness and emotion for any ERP component, but showed, however, large main effects for both factors. Specifically, fearful faces and decreasing face naturalness elicited substantially enlarged N170 and early posterior negativity amplitudes and lower face naturalness also resulted in a larger P1.This pattern reversed for the LPP, showing linear increases in LPP amplitudes with increasing naturalness. We observed no interaction of emotion with face naturalness, which suggests that face naturalness and emotion are decoded in parallel at these early stages. Researchers interested in strong modulations of early components should make use of cutout grey-scaled faces, while those interested in a pronounced late positivity should use close-up coloured faces.

Electrophysiological evidence of altered facial expressions recognition in Alzheimer's disease: A comprehensive ERP study

OBJECTIVES

The present study aims to evaluate the amplitude and latency of event-related potentials (ERPs) P100, N170, VPP and N230 in individuals with Alzheimer's disease (AD) compared to healthy elderly controls, using a passive viewing task of emotional facial expressions.

METHODS

Twenty-four individuals with mild to moderate AD and 23 demographically matched healthy elderly controls were included in the study. ERP P100, N170, VPP and N230 amplitude and latency values were compared between groups.

RESULTS

The categorization of emotional facial expressions was intact; yet, increased P100 amplitude and latency, decreased N170 amplitude, and increased VPP amplitude were observed in AD compared to controls. Increased N230 amplitude and latency were observed in response to angry expressions, while neutral expressions elicited decreased amplitude and latency.

CONCLUSIONS

Increased P100 amplitude and latency may reflect reduced amygdala volume and disruptions in the visual system, while decreased N170 and increased VPP amplitudes may reflect impaired perceptual processing, mitigated by a greater involvement of prefrontal areas for task performance in AD.

SIGNIFICANCE

This study is the first to report a complex pattern of ERPs to emotional facial expressions in individuals with AD.

Component-related BOLD response to localize epileptic focus using simultaneous EEG-fMRI recordings at 3T

BACKGROUND

Simultaneous EEG-fMRI experiments record spatiotemporal dynamics of epileptic activity. A shortcoming of spike-based EEG-fMRI studies is their inability to provide information about behavior of epileptic generators when no spikes are visible.

NEW METHOD

We extract time series of epileptic components identified on EEG and fit them with Generalized Linear Model (GLM) model. This allows a precise and reliable localization of epileptic foci in addition to predicting generator's behavior. The proposed method works in the source domain and delineates generators considering spatial correlation between spike template and candidate components in addition to patient's medical records.

RESULTS

The proposed method was applied on 20 patients with refractory epilepsy and 20 age- and gender-matched healthy controls. The identified components were examined statistically and threshold of localization accuracy was determined as 86% based on Receiver Operating Characteristic (ROC) curve analysis. Accuracy, sensitivity, and specificity were found to be 88%, 85%, and 95%, respectively. Contribution of EEG-fMRI and concordance between EEG and fMRI were also evaluated. Concordance was found in 19 patients and contribution in 17.

COMPARISON WITH EXISTING METHODS

We compared the proposed method with conventional methods. Our comparisons showed superiority of the proposed method. In particular, when epileptogenic zone was located deep in the brain, the method outperformed existing methods.

CONCLUSIONS

This study contributes substantially to increasing the yield of EEG-fMRI and presents a realistic estimate of the neural behavior of epileptic generators, to the best of our knowledge, for the first time in the literature.

The influence of top-down modulation on the processing of direct gaze

Gaze or eye contact is one of the most important nonverbal social cues, which is fundamental to human social interactions. To achieve real time and dynamic face-to-face communication, our brain needs to process another person's gaze direction rapidly and without explicit instruction. In order to explain the fast and spontaneous processing of direct gaze, the fast-track modulator model was proposed. Here, we review recent developments in gaze processing research in the last decade to extend the fast-track modulator model. In particular, we propose that task demand or top-down modulation could play a more crucial role at gaze processing than formerly assumed. We suggest that under different task demands, top-down modulation can facilitate or interfere with the direct gaze effects for early visual processing. The proposed modification of the model extends the role of task demand and its implication on the direct gaze effect, as well as the need to better control for top-down processing in order to better disentangle the role of top-down and bottom-up processing on the direct gaze effect. This article is categorized under: Cognitive Biology > Evolutionary Roots of Cognition Psychology > Perception and Psychophysics Neuroscience > Cognition.