Sustained neural activity to gaze and emotion perception in dynamic social scenes

From Motion to Emotion: Visual Pathways and Potential Interconnections

The two visual pathway description of [Ungerleider, L. G., & Mishkin, M. Two cortical visual systems. In D. J. Dingle, M. A. Goodale, & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549-586). Cambridge, MA: MIT, 1982] changed the course of late 20th century systems and cognitive neuroscience. Here, I try to reexamine our laboratory's work through the lens of the [Pitcher, D., & Ungerleider, L. G. Evidence for a third visual pathway specialized for social perception. Trends in Cognitive Sciences, 25, 100-110, 2021] new third visual pathway. I also briefly review the literature related to brain responses to static and dynamic visual displays, visual stimulation involving multiple individuals, and compare existing models of social information processing for the face and body. In this context, I examine how the posterior STS might generate unique social information relative to other brain regions that also respond to social stimuli. I discuss some of the existing challenges we face with assessing how information flow progresses between structures in the proposed functional pathways and how some stimulus types and experimental designs may have complicated our data interpretation and model generation. I also note a series of outstanding questions for the field. Finally, I examine the idea of a potential expansion of the third visual pathway, to include aspects of previously proposed "lateral" visual pathways. Doing this would yield a more general entity for processing motion/action (i.e., "[inter]action") that deals with interactions between people, as well as people and objects. In this framework, a brief discussion of potential hemispheric biases for function, and different forms of neuropsychological impairments created by focal lesions in the posterior brain is highlighted to help situate various brain regions into an expanded [inter]action pathway.

I can see it in your eyes: Perceived gaze direction impacts ERP and behavioural measures of affective theory of mind

Looking at someone's eyes is thought to be important for affective theory of mind (aTOM), our ability to infer their emotional state. However, it is unknown whether an individual's gaze direction influences our aTOM judgements and what the time course of this influence might be. We presented participants with sentences describing individuals in positive, negative or neutral scenarios, followed by direct or averted gaze neutral face pictures of those individuals. Participants made aTOM judgements about each person's mental state, including their affective valence and arousal, and we investigated whether the face gaze direction impacted those judgements. Participants rated that gazers were feeling more positive when they displayed direct gaze as opposed to averted gaze, and that they were feeling more aroused during negative contexts when gaze was averted as opposed to direct. Event-related potentials associated with face perception and affective processing were examined using mass-univariate analyses to track the time-course of this eye-gaze and affective processing interaction at a neural level. Both positive and negative trials were differentiated from neutral trials at many stages of processing. This included the early N200 and EPN components, believed to reflect automatic emotion areas activation and attentional selection respectively. This also included the later P300 and LPP components, thought to reflect elaborative cognitive appraisal of emotional content. Critically, sentence valence and gaze direction interacted over these later components, which may reflect the incorporation of eye-gaze in the cognitive evaluation of another's emotional state. The results suggest that gaze perception directly impacts aTOM processes, and that altered eye-gaze processing in clinical populations may contribute to associated aTOM impairments.

Visual Information Routes in the Posterior Dorsal and Ventral Face Network Studied with Intracranial Neurophysiology and White Matter Tract Endpoints

Occipitotemporal regions within the face network process perceptual and socioemotional information, but the dynamics and information flow between different nodes of this network are still debated. Here, we analyzed intracerebral EEG from 11 epileptic patients viewing a stimulus sequence beginning with a neutral face with direct gaze. The gaze could avert or remain direct, while the emotion changed to fearful or happy. N200 field potential peak latencies indicated that face processing begins in inferior occipital cortex and proceeds anteroventrally to fusiform and inferior temporal cortices, in parallel. The superior temporal sulcus responded preferentially to gaze changes with augmented field potential amplitudes for averted versus direct gaze, and large effect sizes relative to other network regions. An overlap analysis of posterior white matter tractography endpoints (from 1066 healthy brains) relative to active intracerebral electrodes in the 11 patients showed likely involvement of both dorsal and ventral posterior white matter pathways. Overall, our data provide new insight into the timing of face and social cue processing in the occipitotemporal brain and anchor the superior temporal cortex in dynamic gaze processing.

From Gaze Perception to Social Cognition: The Shared-Attention System

When two people look at the same object in the environment and are aware of each other's attentional state, they find themselves in a shared-attention episode. This can occur through intentional or incidental signaling and, in either case, causes an exchange of information between the two parties about the environment and each other's mental states. In this article, we give an overview of what is known about the building blocks of shared attention (gaze perception and joint attention) and focus on bringing to bear new findings on the initiation of shared attention that complement knowledge about gaze following and incorporate new insights from research into the sense of agency. We also present a neurocognitive model, incorporating first-, second-, and third-order social cognitive processes (the shared-attention system, or SAS), building on previous models and approaches. The SAS model aims to encompass perceptual, cognitive, and affective processes that contribute to and follow on from the establishment of shared attention. These processes include fundamental components of social cognition such as reward, affective evaluation, agency, empathy, and theory of mind.

Naturalistic Stimuli in Neuroscience: Critically Acclaimed

Cognitive neuroscience has traditionally focused on simple tasks, presented sparsely and using abstract stimuli. While this approach has yielded fundamental insights into functional specialisation in the brain, its ecological validity remains uncertain. Do these tasks capture how brains function 'in the wild', where stimuli are dynamic, multimodal, and crowded? Ecologically valid paradigms that approximate real life scenarios, using stimuli such as films, spoken narratives, music, and multiperson games emerged in response to these concerns over a decade ago. We critically appraise whether this approach has delivered on its promise to deliver new insights into brain function. We highlight the challenges, technological innovations, and clinical opportunities that are required should this field meet its full potential.

Automatic Processing of Changes in Facial Emotions in Dysphoria: A Magnetoencephalography Study

It is not known to what extent the automatic encoding and change detection of peripherally presented facial emotion is altered in dysphoria. The negative bias in automatic face processing in particular has rarely been studied. We used magnetoencephalography (MEG) to record automatic brain responses to happy and sad faces in dysphoric (Beck's Depression Inventory ≥ 13) and control participants. Stimuli were presented in a passive oddball condition, which allowed potential negative bias in dysphoria at different stages of face processing (M100, M170, and M300) and alterations of change detection (visual mismatch negativity, vMMN) to be investigated. The magnetic counterpart of the vMMN was elicited at all stages of face processing, indexing automatic deviance detection in facial emotions. The M170 amplitude was modulated by emotion, response amplitudes being larger for sad faces than happy faces. Group differences were found for the M300, and they were indexed by two different interaction effects. At the left occipital region of interest, the dysphoric group had larger amplitudes for sad than happy deviant faces, reflecting negative bias in deviance detection, which was not found in the control group. On the other hand, the dysphoric group showed no vMMN to changes in facial emotions, while the vMMN was observed in the control group at the right occipital region of interest. Our results indicate that there is a negative bias in automatic visual deviance detection, but also a general change detection deficit in dysphoria.

Modulation of Neural Oscillatory Activity during Dynamic Face Processing

Various neuroimaging and neurophysiological methods have been used to examine neural activation patterns in response to faces. However, much of previous research has relied on static images of faces, which do not allow a complete description of the temporal structure of face-specific neural activities to be made. More recently, insights are emerging from fMRI studies about the neural substrates that underpin our perception of naturalistic dynamic face stimuli, but the temporal and spectral oscillatory activity associated with processing dynamic faces has yet to be fully characterized. Here, we used MEG and beamformer source localization to examine the spatiotemporal profile of neurophysiological oscillatory activity in response to dynamic faces. Source analysis revealed a number of regions showing enhanced activation in response to dynamic relative to static faces in the distributed face network, which were spatially coincident with regions that were previously identified with fMRI. Furthermore, our results demonstrate that perception of realistic dynamic facial stimuli activates a distributed neural network at varying time points facilitated by modulations in low-frequency power within alpha and beta frequency ranges (8-30 Hz). Naturalistic dynamic face stimuli may provide a better means of representing the complex nature of perceiving facial expressions in the real world, and neural oscillatory activity can provide additional insights into the associated neural processes.

Processing of gaze direction within the N170/M170 time window: A combined EEG/MEG study

Gaze direction is an important social signal for human beings. Beside the role of gaze in attention orienting, direct gaze (that is, gaze directed toward an observer) is a highly relevant biological stimulus that elicits attention capture and increases face encoding. Brain imaging studies have emphasized the role of the superior temporal sulcus (STS) in the coding of gaze direction and in the integration of gaze and head cues of social attention. The dynamics of the processing and integration of these cues remains, however, unclear. In order to address this question, we used deviated and frontal faces with averted and direct gaze in a combined electro- and magneto- encephalography (EEG-MEG) study. We showed distinct effects of gaze direction on the N170 and M170 responses. There was an interaction between gaze direction and head orientation between 134 and 162ms in MEG and a main effect of gaze direction between 171 and 186ms in EEG. These effects involved the posterior and anterior regions of the STS respectively. Both effects also emphasized the sensitivity to direct gaze. These data highlight the central role of the STS in gaze processing.

Facing two faces: Defense activation varies as a function of personal relevance

It can be unsettling to be watched by a group of people, and when they express anger or hostility, this can prime defensive behavior. In contrast, when others smile at us, this may be comforting. This study tested to which degree the impact of facial expressions (happy, neutral, and angry) varies with the personal relevance of a social situation. Modelling a triadic situation, two faces looked either directly at the participant, faced each other, or they were back to back. Results confirmed that this variation constitutes a gradient of personal relevance (directed frontally > towards > away), as reflected by corresponding defensive startle modulation and autonomic nervous system activity. This gradient was particularly pronounced for angry faces and it was steeper in participants with higher levels of social anxiety. Thus, sender-recipient constellations modulate the processing of facial emotions in favor of adequate behavioral responding (e.g., avoidance) in group settings.

Early Left Parietal Activity Elicited by Direct Gaze: A High-Density EEG Study

Gaze is one of the most important cues for human communication and social interaction. In particular, gaze contact is the most primary form of social contact and it is thought to capture attention. A very early-differentiated brain response to direct versus averted gaze has been hypothesized. Here, we used high-density electroencephalography to test this hypothesis. Topographical analysis allowed us to uncover a very early topographic modulation (40-80 ms) of event-related responses to faces with direct as compared to averted gaze. This modulation was obtained only in the condition where intact broadband faces-as opposed to high-pass or low-pas filtered faces-were presented. Source estimation indicated that this early modulation involved the posterior parietal region, encompassing the left precuneus and inferior parietal lobule. This supports the idea that it reflected an early orienting response to direct versus averted gaze. Accordingly, in a follow-up behavioural experiment, we found faster response times to the direct gaze than to the averted gaze broadband faces. In addition, classical evoked potential analysis showed that the N170 peak amplitude was larger for averted gaze than for direct gaze. Taken together, these results suggest that direct gaze may be detected at a very early processing stage, involving a parallel route to the ventral occipito-temporal route of face perceptual analysis.

Amygdala processing of social cues from faces: an intracrebral EEG study

The amygdala is a key structure for monitoring the relevance of environmental stimuli. Yet, little is known about the dynamics of its response to primary social cues such as gaze and emotion. Here, we examined evoked amygdala responses to gaze and facial emotion changes in five epileptic patients with intracerebral electrodes. Patients first viewed a neutral face that would then convey social cues: it turned either happy or fearful with or without gaze aversion. This social cue was followed by a laterally presented target, the detection of which was faster if it appeared in a location congruent with the averted gaze direction. First, we observed pronounced evoked amygdala potentials to the initial neutral face. Second, analysis of the evoked responses to the cue showed an early effect of gaze starting at 123 ms in the right amygdala. Differential effects of fearful vs happy valence were individually present but more variable in time and therefore not observed at group-level. Our study is the first to demonstrate such an early effect of gaze in the amygdala, in line with its particular behavioral relevance in the spatial attention task.

Social decisions affect neural activity to perceived dynamic gaze

Gaze direction, a cue of both social and spatial attention, is known to modulate early neural responses to faces e.g. N170. However, findings in the literature have been inconsistent, likely reflecting differences in stimulus characteristics and task requirements. Here, we investigated the effect of task on neural responses to dynamic gaze changes: away and toward transitions (resulting or not in eye contact). Subjects performed, in random order, social (away/toward them) and non-social (left/right) judgment tasks on these stimuli. Overall, in the non-social task, results showed a larger N170 to gaze aversion than gaze motion toward the observer. In the social task, however, this difference was no longer present in the right hemisphere, likely reflecting an enhanced N170 to gaze motion toward the observer. Our behavioral and event-related potential data indicate that performing social judgments enhances saliency of gaze motion toward the observer, even those that did not result in gaze contact. These data and that of previous studies suggest two modes of processing visual information: a 'default mode' that may focus on spatial information; a 'socially aware mode' that might be activated when subjects are required to make social judgments. The exact mechanism that allows switching from one mode to the other remains to be clarified.

Magnetoencephalographic study on facial movements

In this review, we introduced our three studies that focused on facial movements. In the first study, we examined the temporal characteristics of neural responses elicited by viewing mouth movements, and assessed differences between the responses to mouth opening and closing movements and an averting eyes condition. Our results showed that the occipitotemporal area, the human MT/V5 homologue, was active in the perception of both mouth and eye motions. Viewing mouth and eye movements did not elicit significantly different activity in the occipitotemporal area, which indicated that perception of the movement of facial parts may be processed in the same manner, and this is different from motion in general. In the second study, we investigated whether early activity in the occipitotemporal region evoked by eye movements was influenced by the facial contour and/or features such as the mouth. Our results revealed specific information processing for eye movements in the occipitotemporal region, and this activity was significantly influenced by whether movements appeared with the facial contour and/or features, in other words, whether the eyes moved, even if the movement itself was the same. In the third study, we examined the effects of inverting the facial contour (hair and chin) and features (eyes, nose, and mouth) on processing for static and dynamic face perception. Our results showed the following: (1) In static face perception, activity in the right fusiform area was affected more by the inversion of features while that in the left fusiform area was affected more by a disruption in the spatial relationship between the contour and features; and (2) In dynamic face perception, activity in the right occipitotemporal area was affected by the inversion of the facial contour.

Neural correlates of apparent motion perception of impoverished facial stimuli: a comparison of ERP and ERSP activity

Our brains readily decode human movements, as shown by neural responses to face and body motion. N170 event-related potentials (ERPs) are earlier and larger to mouth opening movements relative to closing in both line-drawn and natural faces, and gaze aversions relative to direct gaze in natural faces (Puce and Perrett, 2003; Puce et al., 2000). Here we extended this work by recording both ERP and oscillatory EEG activity (event-related spectral perturbations, ERSPs) to line-drawn faces depicting eye and mouth movements (Eyes: Direct vs Away; Mouth: Closed vs Open) and non-face motion controls. Neural activity was measured in 2 occipito-temporal clusters of 9 electrodes, one in each hemisphere. Mouth opening generated larger N170s than mouth closing, replicating earlier work. Eye motion elicited robust N170s that did not differ between gaze conditions. Control condition differences were seen, and generated the largest N170. ERSP difference plots across conditions in the occipito-temporal electrode clusters (Eyes: Direct vs Away; Mouth: Closed vs Open) showed statistically significant differences in beta and gamma bands for gaze direction changes and mouth opening at similar post-stimulus times and frequencies. In contrast, control stimuli showed activity in the gamma band with a completely different time profile and hemispheric distribution to facial stimuli. ERSP plots were generated in two 9 electrode clusters centered on central sites, C3 and C4. In the left cluster for all stimulus conditions, broadband beta suppression persisted from about 250ms post-motion onset. In the right cluster, beta suppression was seen for control conditions only. Statistically significant differences between conditions were confined between 4 and 15Hz, unlike the occipito-temporal sites where differences occurred at much higher frequencies (high beta/gamma). Our data indicate that N170 amplitude is sensitive to the amount of movement in the visual field, independent of stimulus type. In contrast, occipito-temporal beta and gamma activity differentiates between facial and non-facial motion. Context and stimulus configuration likely plays a role in shaping neural responses, based on comparisons of the current data to previously reported studies. Broadband suppression of central beta activity, and significant low frequency differences were likely stimulus driven and not contingent on behavioral responses.

Revealing the neural networks associated with processing of natural social interaction and the related effects of actor-orientation and face-visibility

Understanding the intentions and desires of those around us is vital for adapting to a dynamic social environment. In this paper, a novel event-related functional Magnetic Resonance Imaging (fMRI) paradigm with dynamic and natural stimuli (2s video clips) was developed to directly examine the neural networks associated with processing of gestures with social intent as compared to nonsocial intent. When comparing social to nonsocial gestures, increased activation in both the mentalizing (or theory of mind) and amygdala networks was found. As a secondary aim, a factor of actor-orientation was included in the paradigm to examine how the neural mechanisms differ with respect to personal engagement during a social interaction versus passively observing an interaction. Activity in the lateral occipital cortex and precentral gyrus was found sensitive to actor-orientation during social interactions. Lastly, by manipulating face-visibility we tested whether facial information alone is the primary driver of neural activation differences observed between social and nonsocial gestures. We discovered that activity in the posterior superior temporal sulcus (pSTS) and fusiform gyrus (FFG) was partially driven by observing facial expressions during social gestures. Altogether, using multiple factors associated with processing of natural social interaction, we conceptually advance our understanding of how social stimuli is processed in the brain and discuss the application of this paradigm to clinical populations where atypical social cognition is manifested as a key symptom.

MEG evidence for dynamic amygdala modulations by gaze and facial emotions

BACKGROUND

Amygdala is a key brain region for face perception. While the role of amygdala in the perception of facial emotion and gaze has been extensively highlighted with fMRI, the unfolding in time of amydgala responses to emotional versus neutral faces with different gaze directions is scarcely known.

METHODOLOGY/PRINCIPAL FINDINGS

Here we addressed this question in healthy subjects using MEG combined with an original source imaging method based on individual amygdala volume segmentation and the localization of sources in the amygdala volume. We found an early peak of amygdala activity that was enhanced for fearful relative to neutral faces between 130 and 170 ms. The effect of emotion was again significant in a later time range (310-350 ms). Moreover, the amygdala response was greater for direct relative averted gaze between 190 and 350 ms, and this effect was selective of fearful faces in the right amygdala.

CONCLUSION

Altogether, our results show that the amygdala is involved in the processing and integration of emotion and gaze cues from faces in different time ranges, thus underlining its role in multiple stages of face perception.