Spatially generalizable representations of facial expressions: Decoding across partial face samples

Decoding the temporal representation of facial expression in face-selective regions

The ability of humans to discern facial expressions in a timely manner typically relies on distributed face-selective regions for rapid neural computations. To study the time course in regions of interest for this process, we used magnetoencephalography (MEG) to measure neural responses participants viewed facial expressions depicting seven types of emotions (happiness, sadness, anger, disgust, fear, surprise, and neutral). Analysis of the time-resolved decoding of neural responses in face-selective sources within the inferior parietal cortex (IP-faces), lateral occipital cortex (LO-faces), fusiform gyrus (FG-faces), and posterior superior temporal sulcus (pSTS-faces) revealed that facial expressions were successfully classified starting from ∼100 to 150 ms after stimulus onset. Interestingly, the LO-faces and IP-faces showed greater accuracy than FG-faces and pSTS-faces. To examine the nature of the information processed in these face-selective regions, we entered with facial expression stimuli into a convolutional neural network (CNN) to perform similarity analyses against human neural responses. The results showed that neural responses in the LO-faces and IP-faces, starting ∼100 ms after the stimuli, were more strongly correlated with deep representations of emotional categories than with image level information from the input images. Additionally, we observed a relationship between the behavioral performance and the neural responses in the LO-faces and IP-faces, but not in the FG-faces and lpSTS-faces. Together, these results provided a comprehensive picture of the time course and nature of information involved in facial expression discrimination across multiple face-selective regions, which advances our understanding of how the human brain processes facial expressions.

Fear in the mind's eye: the neural correlates of differential fear acquisition to imagined conditioned stimuli

Mental imagery is involved in both the expression and treatment of fear-related disorders such as anxiety and post-traumatic stress disorder. However, the neural correlates associated with the acquisition and generalization of differential fear conditioning to imagined conditioned stimuli are relatively unknown. In this study, healthy human participants (n = 27) acquired differential fear conditioning to imagined conditioned stimuli paired with a physical unconditioned stimulus (i.e. mild shock), as measured via self-reported fear, the skin conductance response and significant right anterior insula (aIn) activation. Multivoxel pattern analysis cross-classification also demonstrated that the pattern of activity in the right aIn during imagery acquisition was quantifiably similar to the pattern produced by standard visual acquisition. Additionally, mental imagery was associated with significant differential fear generalization. Fear conditioning acquired to imagined stimuli generalized to viewing those same stimuli as measured with self-reported fear and right aIn activity, and likewise fear conditioning to visual stimuli was associated with significant generalized differential self-reported fear and right aIn activity when imagining those stimuli. Together, the study provides a novel understanding of the neural mechanisms associated with the acquisition of differential fear conditioning to imagined stimuli and that of the relationship between imagery and emotion more generally.

The medial temporal lobe structure and function support positive affect

Positive affect (PA) is not only associated with individuals' psychological and physical health, but also their cognitive processes. However, whether medial temporal lobe (MTL) and its subfields' volume/functional connectivity can explain individual variability in PA remains understudied. We investigated the morphological (i.e., grey matter volume; GMV) and functional characteristics (i.e., resting-state functional connectivity; rsFC) of PA with a combination of univariate and multivariate pattern analyses (MVPA) using a large sample of participants (n = 321). We simultaneously collected the T1-weighted (n = 321), high-resolution MTL T2-weighted, and resting-state functional imaging data (n = 209). The MTL and its subfields' volumes, including the CA1, CA2+3, DG, and subiculum (SUB), perirhinal cortex (PRC), and parahippocampus (PHC), were extracted using an automatic segmentation of hippocampal subfields (ASHS) software. The morphological results revealed that GMVs in the prefrontal-occipital and limbic (i.e., hippocampus, amygdala, and PHC) systems were associated with variability in PA at the whole-brain level using MVPA but not univariate analysis. Linear regression results further revealed a positive association between the MTL subfields' GMV, especially for the right PRC, and PA after controlling for several covariates. PRC-seed-based rsFC analyses further revealed that its couplings with the fronto-parietal-occipital system predicted PA in both univariate and MVPA. These findings provide novel insights into the neuroanatomical and functional substrates underlying human PA trait. Findings also suggest critical contributions of the MTL and its subfield of the perirhinal cortex, but not hippocampal subfields, as well as its functional coupling with the fronto-parietal control-system on the formation of PA.

Representational structure of fMRI/EEG responses to dynamic facial expressions

Face perception provides an excellent example of how the brain processes nuanced visual differences and transforms them into behaviourally useful representations of identities and emotional expressions. While a body of literature has looked into the spatial and temporal neural processing of facial expressions, few studies have used a dimensionally varying set of stimuli containing subtle perceptual changes. In the current study, we used 48 short videos varying dimensionally in their intensity and category (happy, angry, surprised) of expression. We measured both fMRI and EEG responses to these video clips and compared the neural response patterns to the predictions of models based on image features and models derived from behavioural ratings of the stimuli. In fMRI, the inferior frontal gyrus face area (IFG-FA) carried information related only to the intensity of the expression, independent of image-based models. The superior temporal sulcus (STS), inferior temporal (IT) and lateral occipital (LO) areas contained information about both expression category and intensity. In the EEG, the coding of expression category and low-level image features were most pronounced at around 400 ms. The expression intensity model did not, however, correlate significantly at any EEG timepoint. Our results show a specific role for IFG-FA in the coding of expressions and suggest that it contains image and category invariant representations of expression intensity.

Decoding sounds depicting hand-object interactions in primary somatosensory cortex

Neurons, even in the earliest sensory regions of cortex, are subject to a great deal of contextual influences from both within and across modality connections. Recent work has shown that primary sensory areas can respond to and, in some cases, discriminate stimuli that are not of their target modality: for example, primary somatosensory cortex (SI) discriminates visual images of graspable objects. In the present work, we investigated whether SI would discriminate sounds depicting hand-object interactions (e.g. bouncing a ball). In a rapid event-related functional magnetic resonance imaging experiment, participants listened attentively to sounds from 3 categories: hand-object interactions, and control categories of pure tones and animal vocalizations, while performing a one-back repetition detection task. Multivoxel pattern analysis revealed significant decoding of hand-object interaction sounds within SI, but not for either control category. Crucially, in the hand-sensitive voxels defined from an independent tactile localizer, decoding accuracies were significantly higher for hand-object interactions compared to pure tones in left SI. Our findings indicate that simply hearing sounds depicting familiar hand-object interactions elicit different patterns of activity in SI, despite the complete absence of tactile stimulation. These results highlight the rich contextual information that can be transmitted across sensory modalities even to primary sensory areas.

Mental imagery can generate and regulate acquired differential fear conditioned reactivity

Mental imagery is an important tool in the cognitive control of emotion. The present study tests the prediction that visual imagery can generate and regulate differential fear conditioning via the activation and prioritization of stimulus representations in early visual cortices. We combined differential fear conditioning with manipulations of viewing and imagining basic visual stimuli in humans. We discovered that mental imagery of a fear-conditioned stimulus compared to imagery of a safe conditioned stimulus generated a significantly greater conditioned response as measured by self-reported fear, the skin conductance response, and right anterior insula activity (experiment 1). Moreover, mental imagery effectively down- and up-regulated the fear conditioned responses (experiment 2). Multivariate classification using the functional magnetic resonance imaging data from retinotopically defined early visual regions revealed significant decoding of the imagined stimuli in V2 and V3 (experiment 1) but significantly reduced decoding in these regions during imagery-based regulation (experiment 2). Together, the present findings indicate that mental imagery can generate and regulate a differential fear conditioned response via mechanisms of the depictive theory of imagery and the biased-competition theory of attention. These findings also highlight the potential importance of mental imagery in the manifestation and treatment of psychological illnesses.

Representation of the brain network by electroencephalograms during facial expressions

BACKGROUND

Facial expressions, such as smiling and anger, cause many physical and psychological effects in the body, known as 'embodied emotions' or 'facial feedback theory.' In the clinical application of this theory in certain diseases, such as autism and depression, treatments such as forcing patients to smile have been used. However, the neural mechanisms underlying the representation of facial expressions remain unclear.

NEW METHOD

We proposed a method to construct brain networks based on the time course of the synchronization likelihood and determine the effects of various facial expressions on the situation using visual stimulus of faces. This method was applied to analyze electroencephalographic (EEG) data recorded during the recognition and representation of various positive and negative facial expressions. The brain networks were constructed based on the EEG data recorded in 11 healthy participants.

RESULTS

Channel sets from brain networks during unsymmetrical smiling expressions (i.e., only the right or left side) were highly linearly symmetrical. Channel sets from brain networks during negative facial expressions (i.e., anger and sadness) and symmetrical smiling expressions (i.e., smiling with an opened or closed mouth) were similar.

COMPARISON WITH EXISTING METHODS

While we obtained brain networks based on time course EEG correlations throughout the experiment, existing methods can analyze EEG data only at a certain time point.

CONCLUSIONS

The comparisons of different facial expressions could be used to identify the side of the facial muscles used while smiling and to determine how similar brain networks are induced by positive and negative facial expressions.

Valence-dependent Disruption in Processing of Facial Expressions of Emotion in Early Visual Cortex—A Transcranial Magnetic Stimulation Study

Our visual inputs are often entangled with affective meanings in natural vision, implying the existence of extensive interaction between visual and emotional processing. However, little is known about the neural mechanism underlying such interaction. This exploratory transcranial magnetic stimulation (TMS) study examined the possible involvement of the early visual cortex (EVC, Area V1/V2/V3) in perceiving facial expressions of different emotional valences. Across three experiments, single-pulse TMS was delivered at different time windows (50–150 msec) after a brief 10-msec onset of face images, and participants reported the visibility and perceived emotional valence of faces. Interestingly, earlier TMS at ∼90 msec only reduced the face visibility irrespective of displayed expressions, but later TMS at ∼120 msec selectively disrupted the recognition of negative facial expressions, indicating the involvement of EVC in the processing of negative expressions at a later time window, possibly beyond the initial processing of fed-forward facial structure information. The observed TMS effect was further modulated by individuals' anxiety level. TMS at ∼110–120 msec disrupted the recognition of anger significantly more for those scoring relatively low in trait anxiety than the high scorers, suggesting that cognitive bias influences the processing of facial expressions in EVC. Taken together, it seems that EVC is involved in structural encoding of (at least) negative facial emotional valence, such as fear and anger, possibly under modulation from higher cortical areas.

The role of imagery in threat-related perceptual decision making

Visual perception is heavily influenced by "top-down" factors, including goals, expectations, and prior knowledge about the environmental context. Recent research has demonstrated the beneficial role threat-related cues play in perceptual decision making; however, the psychological processes contributing to this differential effect remain unclear. Since visual imagery helps to create perceptual representations or "templates" based on prior knowledge (e.g., cues), the present study examines the role vividness of visual imagery plays in enhanced perceptual decision making following threatening cues. In a perceptual decision-making task, participants used threat-related and neutral cues to detect perceptually degraded fearful and neutral faces presented at predetermined perceptual thresholds. Participants' vividness of imagery was measured by the Vividness of Visual Imagery Questionnaire-2 (VVIQ-2). Our results replicated prior work demonstrating that threat cues improve accuracy, perceptual sensitivity, and speed of perceptual decision making compared to neutral cues. Furthermore, better performance following threat and neutral cues was associated with higher VVIQ-2 scores. Importantly, more precise and rapid perceptual decision making following threatening cues was associated with greater VVIQ-2 scores, even after controlling for performance related to neutral cues. This association may be because greater imagery ability allows one to conjure more vivid threat-related templates, which facilitate subsequent perception. While the detection of threatening stimuli is well studied in the literature, our findings elucidate how threatening cues occurring prior to the stimulus aid in subsequent perception. Overall, these findings highlight the necessity of considering top-down threat-related factors in visual perceptual decision making. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Decoding the dynamic representation of facial expressions of emotion in explicit and incidental tasks

Faces transmit a wealth of important social signals. While previous studies have elucidated the network of cortical regions important for perception of facial expression, and the associated temporal components such as the P100, N170 and EPN, it is still unclear how task constraints may shape the representation of facial expression (or other face categories) in these networks. In the present experiment, we used Multivariate Pattern Analysis (MVPA) with EEG to investigate the neural information available across time about two important face categories (expression and identity) when those categories are either perceived under explicit (e.g. decoding facial expression category from the EEG when task is on expression) or incidental task contexts (e.g. decoding facial expression category from the EEG when task is on identity). Decoding of both face categories, across both task contexts, peaked in time-windows spanning 91-170 ms (across posterior electrodes). Peak decoding of expression, however, was not affected by task context whereas peak decoding of identity was significantly reduced under incidental processing conditions. In addition, errors in EEG decoding correlated with errors in behavioral categorization under explicit processing for both expression and identity, however under incidental conditions only errors in EEG decoding of expression correlated with behavior. Furthermore, decoding time-courses and the spatial pattern of informative electrodes showed consistently better decoding of identity under explicit conditions at later-time periods, with weak evidence for similar effects for decoding of expression at isolated time-windows. Taken together, these results reveal differences and commonalities in the processing of face categories under explicit Vs incidental task contexts and suggest that facial expressions are processed to a richer degree under incidental processing conditions, consistent with prior work indicating the relative automaticity by which emotion is processed. Our work further demonstrates the utility in applying multivariate decoding analyses to EEG for revealing the dynamics of face perception.

Identifying and detecting facial expressions of emotion in peripheral vision

Facial expressions of emotion are signals of high biological value. Whilst recognition of facial expressions has been much studied in central vision, the ability to perceive these signals in peripheral vision has only seen limited research to date, despite the potential adaptive advantages of such perception. In the present experiment, we investigate facial expression recognition and detection performance for each of the basic emotions (plus neutral) at up to 30 degrees of eccentricity. We demonstrate, as expected, a decrease in recognition and detection performance with increasing eccentricity, with happiness and surprised being the best recognized expressions in peripheral vision. In detection however, while happiness and surprised are still well detected, fear is also a well detected expression. We show that fear is a better detected than recognized expression. Our results demonstrate that task constraints shape the perception of expression in peripheral vision and provide novel evidence that detection and recognition rely on partially separate underlying mechanisms, with the latter more dependent on the higher spatial frequency content of the face stimulus.