Neurocognitive mechanisms of gaze-expression interactions in face processing and social attention

Electroen cephalography correlates of word and non-word listening in children with specific language impairment: An observational study20F0

Auditory processing in children diagnosed with speech and language impairment (SLI) is atypical and characterized by reduced brain activation compared to typically developing (TD) children. In typical speech and language development processes, frontal, temporal, and posterior regions are engaged during single-word listening, while for non-word listening, it is highly unlikely that perceiving or speaking them is not followed by frequent neurones' activation enough to form stable network connections. This study aimed to investigate the electrophysiological cortical activity of alpha rhythm while listening words and non-words in children with SLI compared to TD children. The participants were 50 children with SLI, aged 4 to 6, and 50 age-related TD children. Groups were divided into 2 subgroups: first subgroup - children aged 4.0 to 5.0 years old (E = 25, C = 25) and second subgroup - children aged 5.0 to 6.0 years old (E = 25, C = 25). The younger children's group did not show statistically significant differences in alpha spectral power in word or non-word listening. In contrast, in the older age group for word and non-word listening, differences were present in the prefrontal, temporal, and parieto-occipital regions bilaterally. Children with SLI showed a certain lack of alpha desynchronization in word and non-word listening compared with TD children. Non-word perception arouses more brain regions because of the unknown presence of the word stimuli. The lack of adequate alpha desynchronization is consistent with established difficulties in lexical and phonological processing at the behavioral level in children with SLI.

How facial masks alter the interaction of gaze direction, head orientation, and emotion recognition

The COVID-19 pandemic has altered the way we interact with each other: mandatory mask-wearing obscures facial information that is crucial for emotion recognition. Whereas the influence of wearing a mask on emotion recognition has been repeatedly investigated, little is known about the impact on interaction effects among emotional signals and other social signals. Therefore, the current study sought to explore how gaze direction, head orientation, and emotional expression interact with respect to emotion perception, and how these interactions are altered by wearing a face mask. In two online experiments, we presented face stimuli from the Radboud Faces Database displaying different facial expressions (anger, fear, happiness, neutral, and sadness), gaze directions (−13°, 0°, and 13°), and head orientations (−45°, 0°, and 45°) – either without (Experiment 1) or with mask (Experiment 2). Participants categorized the displayed emotional expressions. Not surprisingly, masks impaired emotion recognition. Surprisingly, without the mask, emotion recognition was unaffected by averted head orientations and only slightly affected by gaze direction. The mask strongly interfered with this ability. The mask increased the influence of head orientation and gaze direction, in particular for the emotions that were poorly recognized with mask. The results suggest that in case of uncertainty due to ambiguity or absence of signals, we seem to unconsciously factor in extraneous information.

The mediating role of gaze patterns in the association of child sleep disturbances and core symptoms of autism spectrum disorder

Children with autism spectrum disorder (ASD) are at high risk for sleep disturbances, but the mechanism underlying the association between sleep disturbances and ASD core symptoms is largely unknown. This study examined the relationship between sleep disturbances and ASD core symptoms, and the mediating role of gaze patterns during the facial emotion recognition (FER) task. The study included 57 children with ASD and 59 age- and intelligence-matched typically developing (TD) controls aged 3-7 years. Parents reported their children's sleep disturbances and ASD core symptoms using the Children's Sleep Habits Questionnaire (CSHQ) and Social Communication Questionnaire (SCQ). Children's gaze patterns during the FER task were recorded by an eye tracking method. We found (1) ASD children had more severe sleep disturbances than TD children; (2) ASD children had atypical gaze patterns and poor FER task performance as determined by lower accuracy and longer reaction time; (3) sleep disturbances were significantly associated with ASD core symptoms of social interaction, communication, and restricted, repetitive and stereotyped patterns of behavior; and (4) atypical gaze patterns partially mediated the association between sleep disturbances and ASD core symptoms. These findings suggest the need for more comprehensive clinical interventions and more effective sleep interventions to improve ASD core symptoms. LAY SUMMARY: Sleep disturbances are very common in children with autism spectrum disorder (ASD). The current study found that sleep disturbances were significantly associated with ASD core symptoms, and gaze patterns during facial emotion recognition task could partially mediate this relationship.

The Neurobiological Correlates of Gaze Perception in Healthy Individuals and Neurologic Patients

The ability to adaptively follow conspecific eye movements is crucial for establishing shared attention and survival. Indeed, in humans, interacting with the gaze direction of others causes the reflexive orienting of attention and the faster object detection of the signaled spatial location. The behavioral evidence of this phenomenon is called gaze-cueing. Although this effect can be conceived as automatic and reflexive, gaze-cueing is often susceptible to context. In fact, gaze-cueing was shown to interact with other factors that characterize facial stimulus, such as the kind of cue that induces attention orienting (i.e., gaze or non-symbolic cues) or the emotional expression conveyed by the gaze cues. Here, we address neuroimaging evidence, investigating the neural bases of gaze-cueing and the perception of gaze direction and how contextual factors interact with the gaze shift of attention. Evidence from neuroimaging, as well as the fields of non-invasive brain stimulation and neurologic patients, highlights the involvement of the amygdala and the superior temporal lobe (especially the superior temporal sulcus (STS)) in gaze perception. However, in this review, we also emphasized the discrepancies of the attempts to characterize the distinct functional roles of the regions in the processing of gaze. Finally, we conclude by presenting the notion of invariant representation and underline its value as a conceptual framework for the future characterization of the perceptual processing of gaze within the STS.

The Bidirectional Social-Cognitive Mechanisms of the Social-Attention Symptoms of Autism

Differences in social attention development begin to be apparent in the 6th to 12th month of development in children with Autism Spectrum Disorder (ASD) and theoretically reflect important elements of its neurodevelopmental endophenotype. This paper examines alternative conceptual views of these early social attention symptoms and hypotheses about the mechanisms involved in their development. One model emphasizes mechanism involved in the spontaneous allocation of attention to faces, or social orienting. Alternatively, another model emphasizes mechanisms involved in the coordination of attention with other people, or joint attention, and the socially bi-directional nature of its development. This model raises the possibility that atypical responses of children to the attention or the gaze of a social partner directed toward themselves may be as important in the development of social attention symptoms as differences in the development of social orienting. Another model holds that symptoms of social attention may be important to early development, but may not impact older individuals with ASD. The alterative model is that the social attention symptoms in infancy (social orienting and joint attention), and social cognitive symptoms in childhood and adulthood share common neurodevelopmental substrates. Therefore, differences in early social attention and later social cognition constitute a developmentally continuous axis of symptom presentation in ASD. However, symptoms in older individuals may be best measured with in vivo measures of efficiency of social attention and social cognition in social interactions rather than the accuracy of response on analog tests used in measures with younger children. Finally, a third model suggests that the social attention symptoms may not truly be a symptom of ASD. Rather, they may be best conceptualized as stemming from differences domain general attention and motivation mechanisms. The alternative argued for here that infant social attention symptoms meet all the criteria of a unique dimension of the phenotype of ASD and the bi-directional phenomena involved in social attention cannot be fully explained in terms of domain general aspects of attention development.

Human face and gaze perception is highly context specific and involves bottom-up and top-down neural processing

This review summarizes human perception and processing of face and gaze signals. Face and gaze signals are important means of non-verbal social communication. The review highlights that: (1) some evidence is available suggesting that the perception and processing of facial information starts in the prenatal period; (2) the perception and processing of face identity, expression and gaze direction is highly context specific, the effect of race and culture being a case in point. Culture affects by means of experiential shaping and social categorization the way in which information on face and gaze is collected and perceived; (3) face and gaze processing occurs in the so-called 'social brain'. Accumulating evidence suggests that the processing of facial identity, facial emotional expression and gaze involves two parallel and interacting pathways: a fast and crude subcortical route and a slower cortical pathway. The flow of information is bi-directional and includes bottom-up and top-down processing. The cortical networks particularly include the fusiform gyrus, superior temporal sulcus (STS), intraparietal sulcus, temporoparietal junction and medial prefrontal cortex.

Visual Information Routes in the Posterior Dorsal and Ventral Face Network Studied with Intracranial Neurophysiology and White Matter Tract Endpoints

Occipitotemporal regions within the face network process perceptual and socioemotional information, but the dynamics and information flow between different nodes of this network are still debated. Here, we analyzed intracerebral EEG from 11 epileptic patients viewing a stimulus sequence beginning with a neutral face with direct gaze. The gaze could avert or remain direct, while the emotion changed to fearful or happy. N200 field potential peak latencies indicated that face processing begins in inferior occipital cortex and proceeds anteroventrally to fusiform and inferior temporal cortices, in parallel. The superior temporal sulcus responded preferentially to gaze changes with augmented field potential amplitudes for averted versus direct gaze, and large effect sizes relative to other network regions. An overlap analysis of posterior white matter tractography endpoints (from 1066 healthy brains) relative to active intracerebral electrodes in the 11 patients showed likely involvement of both dorsal and ventral posterior white matter pathways. Overall, our data provide new insight into the timing of face and social cue processing in the occipitotemporal brain and anchor the superior temporal cortex in dynamic gaze processing.

The Sexual OBjectification and EMotion database: A free stimulus set and norming data of sexually objectified and non-objectified female targets expressing multiple emotions

Sexual objectification - perceiving or treating a woman as a sexual object - is a widespread phenomenon. Studies on sexual objectification and its consequences have grown dramatically over the last decades covering multiple and diverse areas of research. However, research studying sexual objectification might have limited internal and external validity due to the lack of a controlled and standardized picture database. Moreover, there is a need to extend this research to other fields including the study of emotions. Therefore, in this paper we introduce the SOBEM Database, a free tool consisting of 280 high-resolution pictures depicting objectified and non-objectified female models expressing a neutral face and three different emotions (happiness, anger, and sadness) with different intensity. We report the validation of this dataset by analyzing results of 134 participants judging pictures on the six basic emotions and on a range of social judgments related to sexual objectification. Results showed how the SOBEM can constitute an appropriate instrument to study both sexual objectification per se and its relation with emotions. This database could therefore become an important instrument able to improve the experimental control in future studies on sexual objectification and to create new links with different fields of research.

A Review of Hyperscanning and Its Use in Virtual Environments

Hyperscanning is a technique which simultaneously records the neural activity of two or more people. This is done using one of several neuroimaging methods, such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The use of hyperscanning has seen a dramatic rise in recent years to monitor social interactions between two or more people. Similarly, there has been an increase in the use of virtual reality (VR) for collaboration, and an increase in the frequency of social interactions being carried out in virtual environments (VE). In light of this, it is important to understand how interactions function within VEs, and how they can be enhanced to improve their quality in a VE. In this paper, we present some of the work that has been undertaken in the field of social neuroscience, with a special emphasis on hyperscanning. We also cover the literature detailing the work that has been carried out in the human–computer interaction domain that addresses remote collaboration. Finally, we present a way forward where these two research domains can be combined to explore how monitoring the neural activity of a group of participants in VE could enhance collaboration among them.

Early integration of affectively contextual information when processing low-intensity fearful faces: Behavioral and electrophysiological evidence

In daily social situations, faces rarely appear in isolation and are often contextual. This study investigated individuals' behavioral and neural responses to facial expressions with different intensities embedded in emotional scenes. Participants were presented with neutral, low-intensity, and prototypical fearful expressions embedded in positive and negative scenes and instructed to categorize the facial expressions as neutral or fearful. The behavioral results showed that neutral and low-intensity fearful expressions embedded in negative scenes were rated as fearful significantly more frequently than the same faces embedded in positive scenes. Event-related potential analyses revealed the time course of the integration of contextual information when processing facial expressions. N170 modulation by scene valence was found for low-intensity and prototypical fearful expressions, showing greater N170 elicited by negative than positive scenes. The results suggest that affective information extracted from contextual scenes influences the processing of facial emotions in the early perceptual encoding stage. Furthermore, the pattern of effect varies for different intensities of expressions.

Turning Heads: The Effects of Face View and Eye Gaze Direction on the Perceived Attractiveness of Expressive Faces

We investigated whether the perceived attractiveness of expressive faces was influenced by head turn and eye gaze towards or away from the observer. In all experiments, happy faces were consistently rated as more attractive than angry faces. A head turn towards the observer, whereby a full-face view was shown, was associated with relatively higher attractiveness ratings when gaze direction was aligned with face view (Experiment 1). However, preference for full-face views of happy faces was not affected by gaze shifts towards or away from the observer (Experiment 2a). In Experiment 3, the relative duration of each face view (front-facing or averted at 15°) during a head turn away or towards the observer was manipulated. There was benefit on attractiveness ratings for happy faces shown for a longer duration from the front view, regardless of the direction of head turn. Our findings support previous studies indicating a preference for positive expressions on attractiveness judgements, which is further enhanced by the front views of faces, whether presented during a head turn or shown statically. In sum, our findings imply a complex interaction between cues of social attention, indicated by the view of the face shown, and reward on attractiveness judgements of unfamiliar faces.

No influence of eye gaze on emotional face processing in the absence of conscious awareness

The human brain has evolved specialised mechanisms to enable the rapid detection of threat cues, including emotional face expressions (e.g., fear and anger). However, contextual cues - such as gaze direction - influence the ability to recognise emotional expressions. For instance, anger paired with direct gaze, and fear paired with averted gaze are more accurately recognised compared to alternate conjunctions of these features. It is argued that this is because gaze direction conveys the relevance and locus of the threat to the observer. Here, we used continuous flash suppression (CFS) to assess whether the modulatory effect of gaze direction on emotional face processing occurs outside of conscious awareness. Previous research using CFS has demonstrated that fearful facial expressions are prioritised by the visual system and gain privileged access to awareness over other expressed emotions. We hypothesised that if the modulatory effects of gaze on emotional face processing occur also at this level, then the gaze-emotion conjunctions signalling self-relevant threat will reach awareness faster than those that do not. We report that fearful faces gain privileged access to awareness over angry faces, but that gaze direction does not modulate this effect. Thus, our findings suggest that previously reported effects of gaze direction on emotional face processing are likely to occur once the face is detected, where the self-relevance and locus of the threat can be consciously appraised.

The effects of concurrent cognitive task load on recognising faces displaying emotion

Two independent experiments (N = 30 and N = 24) investigated the effects of concurrent task loads on the recognition of faces displaying emotions. The study aimed to explore a possible resolution for an apparently discrepant finding in the literature regarding the impact of such loads on recognition of facial emotions. Faces displaying different emotions were presented, with or without a concurrent load, until the facial stimuli were correctly labelled to criterion in terms of the displayed emotion. Participants were then presented with elements from the faces (i.e. eyebrows, eyes, and mouth). When participants had to complete the concurrent task as well as the facial recognition task, they did not respond equally to the separate facial elements, and over-selected to the mouth when recognising facial expressions of emotion. The findings relating to the impact of the concurrent load tasks on correct labelling of the facial elements with respect to the emotional faces are discussed in terms of the impact of cognitive load on the production of over-selectivity and the recognition of faces displaying emotions in complex situations, and the implications for those with a developmental disability.

Autism Pathogenesis: The Superior Colliculus

After been exposed to the visual input, in the first year of life, the brain experiences subtle but massive changes apparently crucial for communicative/emotional and social human development. Its lack could be the explanation of the very high prevalence of autism in children with total congenital blindness. The present theory postulates that the superior colliculus is the key structure for such changes for several reasons: it dominates visual behavior during the first months of life; it is ready at birth for complex visual tasks; it has a significant influence on several hemispheric regions; it is the main brain hub that permanently integrates visual and non-visual, external and internal information (bottom-up and top-down respectively); and it owns the enigmatic ability to take non-conscious decisions about where to focus attention. It is also a sentinel that triggers the subcortical mechanisms which drive social motivation to follow faces from birth and to react automatically to emotional stimuli. Through indirect connections it also activates simultaneously several cortical structures necessary to develop social cognition and to accomplish the multiattentional task required for conscious social interaction in real life settings. Genetic or non-genetic prenatal or early postnatal factors could disrupt the SC functions resulting in autism. The timing of postnatal biological disruption matches the timing of clinical autism manifestations. Astonishing coincidences between etiologies, clinical manifestations, cognitive and pathogenic autism theories on one side and SC functions on the other are disclosed in this review. Although the visual system dependent of the SC is usually considered as accessory of the LGN canonical pathway, its imprinting gives the brain a qualitatively specific functions not supplied by any other brain structure.

Face-to-face: Perceived personal relevance amplifies face processing

The human face conveys emotional and social information, but it is not well understood how these two aspects influence face perception. In order to model a group situation, two faces displaying happy, neutral or angry expressions were presented. Importantly, faces were either facing the observer, or they were presented in profile view directed towards, or looking away from each other. In Experiment 1 (n = 64), face pairs were rated regarding perceived relevance, wish-to-interact, and displayed interactivity, as well as valence and arousal. All variables revealed main effects of facial expression (emotional > neutral), face orientation (facing observer > towards > away) and interactions showed that evaluation of emotional faces strongly varies with their orientation. Experiment 2 (n = 33) examined the temporal dynamics of perceptual-attentional processing of these face constellations with event-related potentials. Processing of emotional and neutral faces differed significantly in N170 amplitudes, early posterior negativity (EPN), and sustained positive potentials. Importantly, selective emotional face processing varied as a function of face orientation, indicating early emotion-specific (N170, EPN) and late threat-specific effects (LPP, sustained positivity). Taken together, perceived personal relevance to the observer-conveyed by facial expression and face direction-amplifies emotional face processing within triadic group situations.

How does gaze direction affect facial processing in social anxiety? -An ERP study

Previous behavioral studies have demonstrated an effect of eye gaze direction on the processing of emotional expressions in adults with social anxiety. However, specific brain responses to the interaction between gaze direction and facial expressions in social anxiety remain unclear. The present study aimed to explore the time course of such interaction using event-related potentials (ERPs) in participants with social anxiety. High socially anxious individuals and low socially anxious individuals were asked to identify the gender of angry or neutral faces with direct or averted gaze while their behavioral performance and electrophysiological data were monitored. We found that identification of angry faces with direct but not averted gaze elicited larger N2 amplitude in high socially anxious individuals compared to low socially anxious individuals, while identification of neutral faces did not produce any gaze modulation effect. Moreover, the N2 was correlated with increased anxiety severity upon exposure to angry faces with direct gaze. Therefore, our results suggest that gaze direction modulates the processing of threatening faces in social anxiety. The N2 component elicited by angry faces with direct gaze could be a state-dependent biomarker of social anxiety and may be an important reference biomarker for social anxiety diagnosis and intervention.

Processing of gaze direction within the N170/M170 time window: A combined EEG/MEG study

Gaze direction is an important social signal for human beings. Beside the role of gaze in attention orienting, direct gaze (that is, gaze directed toward an observer) is a highly relevant biological stimulus that elicits attention capture and increases face encoding. Brain imaging studies have emphasized the role of the superior temporal sulcus (STS) in the coding of gaze direction and in the integration of gaze and head cues of social attention. The dynamics of the processing and integration of these cues remains, however, unclear. In order to address this question, we used deviated and frontal faces with averted and direct gaze in a combined electro- and magneto- encephalography (EEG-MEG) study. We showed distinct effects of gaze direction on the N170 and M170 responses. There was an interaction between gaze direction and head orientation between 134 and 162ms in MEG and a main effect of gaze direction between 171 and 186ms in EEG. These effects involved the posterior and anterior regions of the STS respectively. Both effects also emphasized the sensitivity to direct gaze. These data highlight the central role of the STS in gaze processing.

Empathy-Related Responses to Depicted People in Art Works

Existing theories of empathic response to visual art works postulate the primacy of automatic embodied reaction to images based on mirror neuron mechanisms. Arguing for a more inclusive concept of empathy-related response and integrating four distinct bodies of literature, we discuss contextual, and personal factors which modulate empathic response to depicted people. We then present an integrative model of empathy-related responses to depicted people in art works. The model assumes that a response to empathy-eliciting figural artworks engages the dynamic interaction of two mutually interlinked sets of processes: socio-affective/cognitive processing, related to the person perception, and esthetic processing, primarily concerned with esthetic appreciation and judgment and attention to non-social aspects of the image. The model predicts that the specific pattern of interaction between empathy-related and esthetic processing is co-determined by several sets of factors: (i) the viewer's individual characteristics, (ii) the context variables (which include various modes of priming by narratives and other images), (iii) multidimensional features of the image, and (iv) aspects of a viewer's response. Finally we propose that the model is implemented by the interaction of functionally connected brain networks involved in socio-cognitive and esthetic processing.

Shared mechanism for emotion processing in adolescents with and without autism

Although, the quest to understand emotional processing in individuals with Autism Spectrum Disorders (ASD) has led to an impressive number of studies, the picture that emerges from this research remains inconsistent. Some studies find that Typically Developing (TD) individuals outperform those with ASD in emotion recognition tasks, others find no such difference. In this paper, we move beyond focusing on potential group differences in behaviour to answer what we believe is a more pressing question: do individuals with ASD use the same mechanisms to process emotional cues? To this end, we rely on model-based analyses of participants’ accuracy during an emotion categorisation task in which displays of anger and fear are paired with direct vs. averted gaze. Behavioural data of 20 ASD and 20 TD adolescents revealed that the ASD group displayed lower overall performance. Yet, gaze direction had a similar impact on emotion categorisation in both groups, i.e. improved accuracy for salient combinations (anger-direct, fear-averted). Critically, computational modelling of participants’ behaviour reveals that the same mechanism, i.e. increased perceptual sensitivity, underlies the contextual impact of gaze in both groups. We discuss the specific experimental conditions that may favour emotion processing and the automatic integration of contextual information in ASD.

Learning from other people's fear: amygdala-based social reference learning in social anxiety disorder

BACKGROUND

Social anxiety disorder involves fear of social objects or situations. Social referencing may play an important role in the acquisition of this fear and could be a key determinant in future biomarkers and treatment pathways. However, the neural underpinnings mediating such learning in social anxiety are unknown. Using event-related functional magnetic resonance imaging, we examined social reference learning in social anxiety disorder. Specifically, would patients with the disorder show increased amygdala activity during social reference learning, and further, following social reference learning, show particularly increased response to objects associated with other people's negative reactions?

METHOD

A total of 32 unmedicated patients with social anxiety disorder and 22 age-, intelligence quotient- and gender-matched healthy individuals responded to objects that had become associated with others' fearful, angry, happy or neutral reactions.

RESULTS

During the social reference learning phase, a significant group × social context interaction revealed that, relative to the comparison group, the social anxiety group showed a significantly greater response in the amygdala, as well as rostral, dorsomedial and lateral frontal and parietal cortices during the social, relative to non-social, referencing trials. In addition, during the object test phase, relative to the comparison group, the social anxiety group showed increased bilateral amygdala activation to objects associated with others' fearful reactions, and a trend towards decreased amygdala activation to objects associated with others' happy and neutral reactions.

CONCLUSIONS

These results suggest perturbed observational learning in social anxiety disorder. In addition, they further implicate the amygdala and dorsomedial prefrontal cortex in the disorder, and underscore their importance in future biomarker developments.

Investigating the Effect of Gaze Cues and Emotional Expressions on the Affective Evaluations of Unfamiliar Faces

People look at what they are interested in, and their emotional expressions tend to indicate how they feel about the objects at which they look. The combination of gaze direction and emotional expression can therefore convey important information about people’s evaluations of the objects in their environment, and can even influence the subsequent evaluations of those objects by a third party, a phenomenon known as the emotional gaze effect. The present study extended research into the effect of emotional gaze cues by investigating whether they affect evaluations of the most important aspect of our social environment–other people–and whether the presence of multiple gaze cues enhances this effect. Over four experiments, a factorial within-subjects design employing both null hypothesis significance testing and a Bayesian statistical analysis replicated previous work showing an emotional gaze effect for objects, but found strong evidence that emotional gaze cues do not affect evaluations of other people, and that multiple, simultaneously presented gaze cues do not enhance the emotional gaze effect for either the evaluations of objects or of people. Overall, our results suggest that emotional gaze cues have a relatively weak influence on affective evaluations, especially of those aspects of our environment that automatically elicit affectively valenced reactions, including other humans.

Qualitative Analysis of Emotions: Fear and Thrill

People can speak, and this provides opportunities to analyze human emotions using perceived experiences communicated via language, as well as through measurement and imaging techniques that are also applicable to other higher animal species. Here I compare four qualitative methodological approaches to test if, and how, thrill depends on fear. I use eight high-risk, high-skill, real-life outdoor adventure recreation activities to provide the test circumstances. I present data from: >4000 person-days of participant observation; interviews with 40 expert practitioners; retrospective autoethnography of 50 critical incidents over 4 decades; and experimental autoethnography of 60 events. Results from different methods are congruent, but different approaches yield different insights. The principal findings are as follows. Individuals differ in their fear and thrill responses. The same individual may have different responses on different occasions. Fear boosts performance, but panic causes paralysis. Anxiety or apprehension prior to a risky action or event differs from fear experienced during the event itself. The intensity of pre-event fear generally increases with the immediacy of risk to life, and time to contemplate that risk. Fear must be faced, assessed and overcome in order to act. Thrill can occur either during or after a high-risk event. Thrill can occur without fear, and fear without thrill. Below a lower threshold of perceived risk, thrill can occur without fear. Between a lower and upper threshold, thrill increases with fear. Beyond the upper threshold, thrill vanishes but fear remains. This there is a sawtooth relation between fear and thrill. Perceived danger generates intense focus and awareness. Fear and other emotions can disappear during intense concentration and focus. Under high risk, the usual emotional sequence is fear before the action or event, then focus during the action or event, then thrill, relief, or triumph afterward. The emotionless state persists only during the most intense concentration. For events long enough to differentiate time within the events, fear and thrill can arise and fade in different fine-scale sequences.

Contextual Valence and Sociality Jointly Influence the Early and Later Stages of Neutral Face Processing

Recent studies have demonstrated that face perception is influenced by emotional contextual information. However, because facial expressions are routinely decoded and understood during social communication, sociality should also be considered-that is, it seems necessary to explore whether emotional contextual effects are influenced by the sociality of contextual information. Furthermore, although one behavioral study has explored the effects of context on selective attention to faces, the exact underlying mechanisms remain unknown. Therefore, the current study investigated how valence and sociality of contextual information influenced the early and later stages of neutral face processing. We first employed an established affective learning procedure, wherein neutral faces were paired with verbal information that differed in valence (negative, neutral) and sociality (social, non-social), to manipulate contextual information. Then, to explore the effects of context on face perception, participants performed a face perception task, while the N170, early posterior negativity (EPN), and late positive potential (LPP) components were measured. Finally, to explore the effects of context on selective attention, participants performed a dot probe task while the N2pc was recorded. The results showed that, in the face perception task, faces paired with negative social information elicited greater EPN and LPP than did faces paired with neutral social information; no differences existed between faces paired with negative and neutral non-social information. In the dot probe task, faces paired with negative social information elicited a more negative N2pc amplitude (indicating attentional bias) than did faces paired with neutral social information; the N2pc did not differ between faces paired with negative and neutral non-social information. Together, these results suggest that contextual information influenced both face perception and selective attention, and these context effects were governed by the interaction between valence and sociality of contextual information.

Interactive effects between gaze direction and facial expression on attentional resources deployment: the task instruction and context matter

In three experiments, we tested whether the amount of attentional resources needed to process a face displaying neutral/angry/fearful facial expressions with direct or averted gaze depends on task instructions, and face presentation. To this end, we used a Rapid Serial Visual Presentation paradigm in which participants in Experiment 1 were first explicitly asked to discriminate whether the expression of a target face (T1) with direct or averted gaze was angry or neutral, and then to judge the orientation of a landscape (T2). Experiment 2 was identical to Experiment 1 except that participants had to discriminate the gender of the face of T1 and fearful faces were also presented randomly inter-mixed within each block of trials. Experiment 3 differed from Experiment 2 only because angry and fearful faces were never presented within the same block. The findings indicated that the presence of the attentional blink (AB) for face stimuli depends on specific combinations of gaze direction and emotional facial expressions and crucially revealed that the contextual factors (e.g., explicit instruction to process the facial expression and the presence of other emotional faces) can modify and even reverse the AB, suggesting a flexible and more contextualized deployment of attentional resources in face processing.

Convergent evidence for top-down effects from the "predictive brain"

Modern conceptions of brain function consider the brain as a "predictive organ," where learned regularities about the world are utilised to facilitate perception of incoming sensory input. Critically, this process hinges on a role for cognitive penetrability. We review a mechanism to explain this process and expand our previous proposals of cognitive penetrability in visual recognition to social vision and visual hallucinations.

Interference with facial emotion recognition by verbal but not visual loads

The ability to recognize emotions through facial characteristics is critical for social functioning, but is often impaired in those with a developmental or intellectual disability. The current experiments explored the degree to which interfering with the processing capacities of typically-developing individuals would produce a similar inability to recognize emotions through the facial elements of faces displaying particular emotions. It was found that increasing the cognitive load (in an attempt to model learning impairments in a typically developing population) produced deficits in correctly identifying emotions from facial elements. However, this effect was much more pronounced when using a concurrent verbal task than when employing a concurrent visual task, suggesting that there is a substantial verbal element to the labeling and subsequent recognition of emotions. This concurs with previous work conducted with those with developmental disabilities that suggests emotion recognition deficits are connected with language deficits.

Facial mimicry in its social setting

In interpersonal encounters, individuals often exhibit changes in their own facial expressions in response to emotional expressions of another person. Such changes are often called facial mimicry. While this tendency first appeared to be an automatic tendency of the perceiver to show the same emotional expression as the sender, evidence is now accumulating that situation, person, and relationship jointly determine whether and for which emotions such congruent facial behavior is shown. We review the evidence regarding the moderating influence of such factors on facial mimicry with a focus on understanding the meaning of facial responses to emotional expressions in a particular constellation. From this, we derive recommendations for a research agenda with a stronger focus on the most common forms of encounters, actual interactions with known others, and on assessing potential mediators of facial mimicry. We conclude that facial mimicry is modulated by many factors: attention deployment and sensitivity, detection of valence, emotional feelings, and social motivations. We posit that these are the more proximal causes of changes in facial mimicry due to changes in its social setting.

Amygdala processing of social cues from faces: an intracrebral EEG study

The amygdala is a key structure for monitoring the relevance of environmental stimuli. Yet, little is known about the dynamics of its response to primary social cues such as gaze and emotion. Here, we examined evoked amygdala responses to gaze and facial emotion changes in five epileptic patients with intracerebral electrodes. Patients first viewed a neutral face that would then convey social cues: it turned either happy or fearful with or without gaze aversion. This social cue was followed by a laterally presented target, the detection of which was faster if it appeared in a location congruent with the averted gaze direction. First, we observed pronounced evoked amygdala potentials to the initial neutral face. Second, analysis of the evoked responses to the cue showed an early effect of gaze starting at 123 ms in the right amygdala. Differential effects of fearful vs happy valence were individually present but more variable in time and therefore not observed at group-level. Our study is the first to demonstrate such an early effect of gaze in the amygdala, in line with its particular behavioral relevance in the spatial attention task.

Social decisions affect neural activity to perceived dynamic gaze

Gaze direction, a cue of both social and spatial attention, is known to modulate early neural responses to faces e.g. N170. However, findings in the literature have been inconsistent, likely reflecting differences in stimulus characteristics and task requirements. Here, we investigated the effect of task on neural responses to dynamic gaze changes: away and toward transitions (resulting or not in eye contact). Subjects performed, in random order, social (away/toward them) and non-social (left/right) judgment tasks on these stimuli. Overall, in the non-social task, results showed a larger N170 to gaze aversion than gaze motion toward the observer. In the social task, however, this difference was no longer present in the right hemisphere, likely reflecting an enhanced N170 to gaze motion toward the observer. Our behavioral and event-related potential data indicate that performing social judgments enhances saliency of gaze motion toward the observer, even those that did not result in gaze contact. These data and that of previous studies suggest two modes of processing visual information: a 'default mode' that may focus on spatial information; a 'socially aware mode' that might be activated when subjects are required to make social judgments. The exact mechanism that allows switching from one mode to the other remains to be clarified.

Selective attention effects on early integration of social signals: same timing, modulated neural sources

Humans combine co-emitted social signals to predict other's immediate intentions and prepare an adapted response. However, little is known about whether attending to only one of co-emitted social signals impacts on its combination with other signals. Here, using electroencephalography, we address selective attention effects on early combination of social signals. We manipulated three visual cues: gaze direction, emotional expression, and pointing gesture, while participants performed either emotion or gaze direction judgments. Results showed that a temporal marker of social cues integration emerges 170ms after the stimulus onset, even if the integration of the three visual cues was not required to perform the task, as only one feature at a time was task relevant. Yet in addition to common temporal regions, the relative contribution of specific neural sources of this integration changed as a function of the attended feature: integration during emotion judgments was mainly implemented in classic limbic areas but in the dorsal pathway during gaze direction judgments. Together, these findings demonstrate that co-emitted social cues are integrated as long as they are relevant to the observer, even when they are irrelevant to the ongoing task.

The influences of face inversion and facial expression on sensitivity to eye contact in high-functioning adults with autism spectrum disorders

We examined the influences of face inversion and facial expression on sensitivity to eye contact in high-functioning adults with and without an autism spectrum disorder (ASD). Participants judged the direction of gaze of angry, fearful, and neutral faces. In the typical group only, the range of directions of gaze leading to the perception of eye contact (the cone of gaze) was narrower for upright than inverted faces. In both groups, the cone of gaze was wider for angry faces than for fearful or neutral faces. These results suggest that in high-functioning adults with ASD, the perception of eye contact is not tuned to be finer for upright than inverted faces, but that information is nevertheless integrated across expression and gaze direction.

Hierarchical Encoding of Social Cues in Primate Inferior Temporal Cortex

Faces convey information about identity and emotional state, both of which are important for our social interactions. Models of face processing propose that changeable versus invariant aspects of a face, specifically facial expression/gaze direction versus facial identity, are coded by distinct neural pathways and yet neurophysiological data supporting this separation are incomplete. We recorded activity from neurons along the inferior bank of the superior temporal sulcus (STS), while monkeys viewed images of conspecific faces and non-face control stimuli. Eight monkey identities were used, each presented with 3 different facial expressions (neutral, fear grin, and threat). All facial expressions were displayed with both a direct and averted gaze. In the posterior STS, we found that about one-quarter of face-responsive neurons are sensitive to social cues, the majority of which being sensitive to only one of these cues. In contrast, in anterior STS, not only did the proportion of neurons sensitive to social cues increase, but so too did the proportion of neurons sensitive to conjunctions of identity with either gaze direction or expression. These data support a convergence of signals related to faces as one moves anteriorly along the inferior bank of the STS, which forms a fundamental part of the face-processing network.

Gaze perception in social anxiety and social anxiety disorder

Clinical observations suggest abnormal gaze perception to be an important indicator of social anxiety disorder (SAD). Experimental research has yet paid relatively little attention to the study of gaze perception in SAD. In this article we first discuss gaze perception in healthy human beings before reviewing self-referential and threat-related biases of gaze perception in clinical and non-clinical socially anxious samples. Relative to controls, socially anxious individuals exhibit an enhanced self-directed perception of gaze directions and demonstrate a pronounced fear of direct eye contact, though findings are less consistent regarding the avoidance of mutual gaze in SAD. Prospects for future research and clinical implications are discussed.

Dissimilar processing of emotional facial expressions in human and monkey temporal cortex

Emotional facial expressions play an important role in social communication across primates. Despite major progress made in our understanding of categorical information processing such as for objects and faces, little is known, however, about how the primate brain evolved to process emotional cues. In this study, we used functional magnetic resonance imaging (fMRI) to compare the processing of emotional facial expressions between monkeys and humans. We used a 2×2×2 factorial design with species (human and monkey), expression (fear and chewing) and configuration (intact versus scrambled) as factors. At the whole brain level, neural responses to conspecific emotional expressions were anatomically confined to the superior temporal sulcus (STS) in humans. Within the human STS, we found functional subdivisions with a face-selective right posterior STS area that also responded to emotional expressions of other species and a more anterior area in the right middle STS that responded specifically to human emotions. Hence, we argue that the latter region does not show a mere emotion-dependent modulation of activity but is primarily driven by human emotional facial expressions. Conversely, in monkeys, emotional responses appeared in earlier visual cortex and outside face-selective regions in inferior temporal cortex that responded also to multiple visual categories. Within monkey IT, we also found areas that were more responsive to conspecific than to non-conspecific emotional expressions but these responses were not as specific as in human middle STS. Overall, our results indicate that human STS may have developed unique properties to deal with social cues such as emotional expressions.

Sustained neural activity to gaze and emotion perception in dynamic social scenes

To understand social interactions, we must decode dynamic social cues from seen faces. Here, we used magnetoencephalography (MEG) to study the neural responses underlying the perception of emotional expressions and gaze direction changes as depicted in an interaction between two agents. Subjects viewed displays of paired faces that first established a social scenario of gazing at each other (mutual attention) or gazing laterally together (deviated group attention) and then dynamically displayed either an angry or happy facial expression. The initial gaze change elicited a significantly larger M170 under the deviated than the mutual attention scenario. At around 400 ms after the dynamic emotion onset, responses at posterior MEG sensors differentiated between emotions, and between 1000 and 2200 ms, left posterior sensors were additionally modulated by social scenario. Moreover, activity on right anterior sensors showed both an early and prolonged interaction between emotion and social scenario. These results suggest that activity in right anterior sensors reflects an early integration of emotion and social attention, while posterior activity first differentiated between emotions only, supporting the view of a dual route for emotion processing. Altogether, our data demonstrate that both transient and sustained neurophysiological responses underlie social processing when observing interactions between others.

Facial expressions of threat influence perceived gaze direction in 8 year-olds

Adults show reciprocal influences between the perception of gaze direction and emotional expression. These facilitate the understanding of facial signals, because the meaning of one cue can vary considerably depending on the value of the other. Here we ask whether children show similar reciprocal influences in the perception of gaze and expression. A previous study has demonstrated that gaze direction affects the perception of emotional expression in children. Here we demonstrate the opposite direction of influence, showing that expression affects the perception of gaze direction. Specifically, we show that the cone of gaze, i.e., range of gaze deviations perceived as direct, is larger for angry than neutral or fearful faces in 8 year-old children. Therefore, we conclude that children, like adults, show reciprocal influences in the perception of gaze and expression. An unexpected finding was that, compared with adults, children showed larger effects of expression on gaze perception. This finding raises the possibility that it is the ability to process cues independently, rather than sensitivity to combinations, that matures during development. Alternatively, children may be particularly sensitive to anger in adult faces.

Faces in context: a review and systematization of contextual influences on affective face processing

Facial expressions are of eminent importance for social interaction as they convey information about other individuals’ emotions and social intentions. According to the predominant “basic emotion” approach, the perception of emotion in faces is based on the rapid, automatic categorization of prototypical, universal expressions. Consequently, the perception of facial expressions has typically been investigated using isolated, de-contextualized, static pictures of facial expressions that maximize the distinction between categories. However, in everyday life, an individual’s face is not perceived in isolation, but almost always appears within a situational context, which may arise from other people, the physical environment surrounding the face, as well as multichannel information from the sender. Furthermore, situational context may be provided by the perceiver, including already present social information gained from affective learning and implicit processing biases such as race bias. Thus, the perception of facial expressions is presumably always influenced by contextual variables. In this comprehensive review, we aim at (1) systematizing the contextual variables that may influence the perception of facial expressions and (2) summarizing experimental paradigms and findings that have been used to investigate these influences. The studies reviewed here demonstrate that perception and neural processing of facial expressions are substantially modified by contextual information, including verbal, visual, and auditory information presented together with the face as well as knowledge or processing biases already present in the observer. These findings further challenge the assumption of automatic, hardwired categorical emotion extraction mechanisms predicted by basic emotion theories. Taking into account a recent model on face processing, we discuss where and when these different contextual influences may take place, thus outlining potential avenues in future research.