How the visual brain detects emotional changes in facial expressions: Evidence from driven and intrinsic brain oscillations

The use of happy faces as visual stimuli improves the performance of the hybrid SSVEP+P300 brain computer interface

BACKGROUND

This study illustrates a hybrid brain-computer interface (BCI) in which steady-state visual evoked potentials (SSVEP) and event-related potentials (P300) are evoked simultaneously. The goal of this study was to improve the performance of the current hybrid SSVEP+P300 BCI systems by incorporating a happy face into visual stimuli.

NEW METHOD

In this study, happy and sad faces were added to a visual stimulus to induce stronger cortical signals in a hybrid SSVEP+P300 BCI. Additionally, we developed a paradigm in which SSVEP responses were triggered by non-face stimuli, whereas P300 responses were triggered by face stimuli. We tested four paradigms: happy face paradigm (HF), sad face paradigm (SF), happy face and flicker paradigm (HFF), and sad face and flicker paradigm (SFF).

RESULTS AND CONCLUSIONS

The results demonstrated that the HFF paradigm elicited more robust cortical responses, which resulted in enhanced system accuracy and information transfer rate (ITR). The HFF paradigm has a system communication rate of 25.9 bits per second and an average accuracy of 96.1%. Compared with other paradigms, the HFF paradigm is the best choice for BCI applications because it has the highest ITR and maximum level of comfort.

Event-related alpha power in early stage of facial expression processing in social anxiety: Influence of language context

Accurate interpretation of the emotional information conveyed by others' facial expressions is crucial for social interactions. Event-related alpha power, measured by time-frequency analysis, is a frequently used EEG index of emotional information processing. However, it is still unclear how event-related alpha power varies in emotional information processing in social anxiety groups. In the present study, we recorded event-related potentials (ERPs) while participants from the social anxiety and healthy control groups viewed facial expressions (angry, happy, neutral) preceded by contextual sentences conveying either a positive or negative evaluation of the subject. The impact of context on facial expression processing in both groups of participants was explored by assessing behavioral ratings and event-related alpha power (0-200 ms after expression presentation). In comparison to the healthy control group, the social anxiety group exhibited significantly lower occipital alpha power in response to angry facial expressions in negative contexts and neutral facial expressions in positive contexts. The influence of language context on facial expression processing in individuals with social anxiety may occur at an early stage of processing.

Neural response and representation: Facial expressions in scenes

Previous studies have shown that the brain generates expectations based on scenes, which affects facial expression recognition. However, although facial expressions are known to interact with perception, the mechanism underlying this interaction remains poorly understood. Here, we used frequency labeling and decoding techniques to reveal the effects of scene-based expectation on the amplitude and representational strength of neural activity. We also reduced the relative reliability between expectation and sensory input by blurring facial expressions to further investigate the effects of this relative reliability on the pattern of neural activation and representation. Participants viewed emotional changes in unblurred or blurred facial expressions, which flickered at a rate of 6 Hz within a scene. We found that facial expressions that were congruent with the emotional significance of the scene elicited a larger steady-state visual evoked potential amplitude than did facial expressions that were incongruent with the emotional significance of a scene, in both unblurred and blurred conditions. We also found that expected facial expression representations were stronger than unexpected representations during the unblurred condition. In the blurred condition, unexpected representations were stronger than expected representations. Taken together, these results suggested that facial expression processing in the visual cortex is modulated by top-down signals. The relative reliability of expectation and sensory input moderated the influence of a scene on facial expression representation. Furthermore, our study showed that neural activation amplitudes did not correspond to representational strength.

Investigating sustained attention in contextual threat using steady‐state VEPs evoked by flickering video stimuli

Anxiety is characterized by anxious anticipation and heightened vigilance to uncertain threat. However, if threat is not reliably indicated by a specific cue, the context in which threat was previously experienced becomes its best predictor, leading to anxiety. A suitable means to induce anxiety experimentally is context conditioning: In one context (CTX+), an unpredictable aversive stimulus (US) is repeatedly presented, in contrast to a second context (CTX-), in which no US is ever presented. In this EEG study, we investigated attentional mechanisms during acquisition and extinction learning in 38 participants, who underwent a context conditioning protocol. Flickering video stimuli (32 s clips depicting virtual offices representing CTX+/-) were used to evoke steady-state visual evoked potentials (ssVEPs) as an index of visuocortical engagement with the contexts. Analyses of the electrocortical responses suggest a successful induction of the ssVEP signal by video presentation in flicker mode. Furthermore, we found clear indices of context conditioning and extinction learning on a subjective level, while cortical processing of the CTX+ was unexpectedly reduced during video presentation. The differences between CTX+ and CTX- diminished during extinction learning. Together, these results indicate that the dynamic sensory input of the video presentation leads to disruptions in the ssVEP signal, which is greater for motivationally significant, threatening contexts.

The rapid and automatic categorization of facial expression changes in highly variable natural images

Emotional expressions are quickly and automatically read from human faces under natural viewing conditions. Yet, categorization of facial expressions is typically measured in experimental contexts with homogenous sets of face stimuli. Here we evaluated how the 6 basic facial emotions (Fear, Disgust, Happiness, Anger, Surprise or Sadness) can be rapidly and automatically categorized with faces varying in head orientation, lighting condition, identity, gender, age, ethnic origin and background context. High-density electroencephalography was recorded in 17 participants viewing 50 s sequences with natural variable images of neutral-expression faces alternating at a 6 Hz rate. Every five stimuli (1.2 Hz), variable natural images of one of the six basic expressions were presented. Despite the wide physical variability across images, a significant F/5 = 1.2 Hz response and its harmonics (e.g., 2F/5 = 2.4 Hz, etc.) was observed for all expression changes at the group-level and in every individual participant. Facial categorization responses were found mainly over occipito-temporal sites, with distinct hemispheric lateralization and cortical topographies according to the different expressions. Specifically, a stronger response was found to Sadness categorization, especially over the left hemisphere, as compared to Fear and Happiness, together with a right hemispheric dominance for categorization of Fearful faces. Importantly, these differences were specific to upright faces, ruling out the contribution of low-level visual cues. Overall, these observations point to robust rapid and automatic facial expression categorization processes in the human brain.

Social aversive generalization learning sharpens the tuning of visuocortical neurons to facial identity cues

Defensive system activation promotes heightened perception of threat signals, and excessive attention to threat signals has been discussed as a contributory factor in the etiology of anxiety disorders. However, a mechanistic account of attentional modulation during fear-relevant processes, especially during fear generalization remains elusive. To test the hypothesis that social fear generalization prompts sharpened tuning in the visuocortical representation of social threat cues, 67 healthy participants underwent differential fear conditioning, followed by a generalization test in which participants viewed faces varying in similarity with the threat-associated face. We found that generalization of social threat sharpens visuocortical tuning of social threat cues, whereas ratings of fearfulness showed generalization, linearly decreasing with decreasing similarity to the threat-associated face. Moreover, individuals who reported greater anxiety in social situations also showed heightened sharpened tuning of visuocortical neurons to facial identity cues, indicating the behavioral relevance of visuocortical tuning during generalization learning.

Affective Bias without Hemispheric Competition: Evidence for Independent Processing Resources in Each Cortical Hemisphere

We assessed the extent of neural competition for attentional processing resources in early visual cortex between foveally presented task stimuli and peripheral emotional distracter images. Task-relevant and distracting stimuli were shown in rapid serial visual presentation (RSVP) streams to elicit the steady-state visual evoked potential, which serves as an electrophysiological marker of attentional resource allocation in early visual cortex. A task-related RSVP stream of symbolic letters was presented centrally at 15 Hz while distracting RSVP streams were displayed at 4 or 6 Hz in the left and right visual hemifields. These image streams always had neutral content in one visual field and would unpredictably switch from neutral to unpleasant content in the opposite visual field. We found that the steady-state visual evoked potential amplitude was consistently modulated as a function of change in emotional valence in peripheral RSVPs, indicating sensory gain in response to distracting affective content. Importantly, the facilitated processing for emotional content shown in one visual hemifield was not paralleled by any perceptual costs in response to the task-related processing in the center or the neutral image stream in the other visual hemifield. Together, our data provide further evidence for sustained sensory facilitation in favor of emotional distracters. Furthermore, these results are in line with previous reports of a “different hemifield advantage” with low-level visual stimuli and are suggestive of independent processing resources in each cortical hemisphere that operate beyond low-level visual cues, that is, with complex images that impact early stages of visual processing via reentrant feedback loops from higher order processing areas.

Neural responses to criticism and praise vary with schizotypy and perceived emotional support

Schizotypy is a latent organisation of a cluster of personality styles, such as magical thinking, disorganisation and anhedonia, which are in the normal range of the psychosis continuum. Schizotypy relates to an increased likelihood of perceiving expressed emotion (EE). EE is characterised by criticism, rejection, and emotional over-involvement and less warmth from a close relative. Neuroimaging studies have found normal frontal lobe activation to EE-criticism in people with high schizotypy. Alternatively, electroencephalography measures emotion processing, such as frontal theta power and occipital alpha power. Frontal theta power responds to cognitive and affective processes and occipital alpha power denotes less consciousness and emotional attention. This study aimed to determine the relation of these electroencephalography responses during criticism and praise to perceived emotional support. Participants (n = 32) representing the full (low-to-high) range of positive schizotypy listened to and rated the self-relevance of EE-like criticism and praise and affectively neutral comments while undergoing electroencephalography. Participants completed self-report measures of schizotypy, depression and anxiety. A subset of those with a high positive schizotypy score (n = 22) completed a measure of perceived EE - lack of emotional support. Higher perceived EE - lack of emotional support correlated with lower frontal theta power and lower occipital alpha power during criticism and praise in schizotypal participants. The findings suggest that these neural responses may relate to less perceived emotional support in people with high schizotypy, of which a reduction of frontal theta power denotes less emotional arousal and lower occipital alpha power denotes more alertness to emotional information may relate to less perceived emotional support in people with high schizotypy.