Interactions between the processing of gaze direction and facial expression

The relationship between ensemble representations of facial information

People accurately evaluate various types of facial information (gaze direction, facial expression, facial identity, and gender) of multiple faces. Considering such varieties, summarizing abilities of facial information might vary depending on its type because it is either changeable (e.g., gaze direction and expression) or invariant (e.g., identity and gender). The current study investigated the relationship between the averaging abilities of facial information using an individual difference approach and a dual-task paradigm to understand the effect of facial information type on the ensemble coding of facial information. We conducted two online experiments on the relationship between the averaging abilities of facial expressions and gaze direction (Experiment 1) and those of facial expressions and gender (Experiment 2). Participants were asked to estimate the average of each piece of facial information in the first and second blocks (single task), respectively, and both sequentially in the third and fourth blocks (dual task). We found that most of the error autocorrelations of facial information were high, indicating high measurement reliability. Participants' abilities to average facial expressions were correlated with those to average gaze directions, but not with those to average gender information. That is, the ensemble processing of facial expressions is related to gaze directions, but not genders. These results suggest that ensemble representations of facial information regarding changeable properties differ from those of invariant ones.

How facial masks alter the interaction of gaze direction, head orientation, and emotion recognition

The COVID-19 pandemic has altered the way we interact with each other: mandatory mask-wearing obscures facial information that is crucial for emotion recognition. Whereas the influence of wearing a mask on emotion recognition has been repeatedly investigated, little is known about the impact on interaction effects among emotional signals and other social signals. Therefore, the current study sought to explore how gaze direction, head orientation, and emotional expression interact with respect to emotion perception, and how these interactions are altered by wearing a face mask. In two online experiments, we presented face stimuli from the Radboud Faces Database displaying different facial expressions (anger, fear, happiness, neutral, and sadness), gaze directions (−13°, 0°, and 13°), and head orientations (−45°, 0°, and 45°) – either without (Experiment 1) or with mask (Experiment 2). Participants categorized the displayed emotional expressions. Not surprisingly, masks impaired emotion recognition. Surprisingly, without the mask, emotion recognition was unaffected by averted head orientations and only slightly affected by gaze direction. The mask strongly interfered with this ability. The mask increased the influence of head orientation and gaze direction, in particular for the emotions that were poorly recognized with mask. The results suggest that in case of uncertainty due to ambiguity or absence of signals, we seem to unconsciously factor in extraneous information.

The role of discriminability in face perception: Interference processing of expression, gender, and gaze

Eye gaze plays a fundamental role in social interaction and facial recognition. However, interference processing between gaze and other facial variants (e.g., expression) and invariant information (e.g., gender) remains controversial and unclear, especially the role of facial information discriminability in interference. A Garner paradigm was used to conduct two experiments. This paradigm allows simultaneous investigation of the mutual influence of two kinds of facial information in one experiment. In Experiment 1, we manipulated facial expression discriminability and investigated its role in interference processing of gaze and facial expression. The results show that individuals were unable to ignore expression when classifying gaze with both high and low discriminability but could ignore gaze when classifying expression with high discriminability only. In Experiment 2, we manipulated gender discriminability and investigated its function in interference processing of gaze and gender. Participants were unable to ignore gender when classifying gaze with both high and low discriminability but could ignore gaze when classifying gender with low discriminability only. The results indicate that gaze categorization is affected by facial expression and gender regardless of facial information discriminability, whereas interference of gaze on facial expression and gender depends on the degree of discriminability. The present study provides evidence that the processing of gaze and other variant and invariant information is interdependent.

Inversion Reduces Sensitivity to Complex Emotions in Eye Regions

Inferring humans’ complex emotions is challenging but can be done with surprisingly limited emotion signals, including merely the eyes alone. Here, we test for a role of lower-level perceptual processes involved in such sensitivity using the well-validated Reading the Mind in the Eyes task. Over three experiments, we manipulated configural processing to show that it contributes to sensitivity to complex emotion from human eye regions. Specifically, inversion, a well-established manipulation affecting configural processing, undermined sensitivity to complex emotions in eye regions (Experiments 1-3). Inversion extended to undermine sensitivity to nonmentalistic information from human eye regions (gender; Experiment 2) but did not extend to affect sensitivity to attributes of nonhuman animals (Experiment 3). Taken together, the current findings provide evidence for the novel hypothesis that configural processing facilitates sensitivity to complex emotions conveyed by the eyes via the broader extraction of socially relevant information.

Age of smile: a cross-cultural replication report of Ganel and Goodale (2018)

Smiling is believed to make people look younger. Ganel and Goodale (Psychon Bull Rev 25(6):612–616, 10.3758/s13423-017-1306-8, 2018) proposed that this belief is a misconception rooted in popular media, based on their findings that people actually perceive smiling faces as older. However, they did not clarify whether this misconception can be generalized across cultures. We tested the cross-cultural validity of Ganel and Goodale’s findings by collecting data from Japanese and Swedish participants. Specifically, we aimed to replicate Ganel and Goodale’s study using segregated sets of Japanese and Swedish facial stimuli, and including Japanese and Swedish participants in groups asked to estimate the age of either Japanese or Swedish faces (two groups of participants × two groups of stimuli; four groups total). Our multiverse analytical approach consistently showed that the participants evaluated smiling faces as older in direct evaluations, regardless of the facial stimuli culture or their nationality, although they believed that smiling makes people look younger. Further, we hypothesized that the effect of wrinkles around the eyes on the estimation of age would vary with the stimulus culture, based on previous studies. However, we found no differences in age estimates by stimulus culture in the present study. Our results showed that we successfully replicated Ganel and Goodale (2018) in a cross-cultural context. Our study thus clarified that the belief that smiling makes people look younger is a common cultural misconception.

Turning Heads: The Effects of Face View and Eye Gaze Direction on the Perceived Attractiveness of Expressive Faces

We investigated whether the perceived attractiveness of expressive faces was influenced by head turn and eye gaze towards or away from the observer. In all experiments, happy faces were consistently rated as more attractive than angry faces. A head turn towards the observer, whereby a full-face view was shown, was associated with relatively higher attractiveness ratings when gaze direction was aligned with face view (Experiment 1). However, preference for full-face views of happy faces was not affected by gaze shifts towards or away from the observer (Experiment 2a). In Experiment 3, the relative duration of each face view (front-facing or averted at 15°) during a head turn away or towards the observer was manipulated. There was benefit on attractiveness ratings for happy faces shown for a longer duration from the front view, regardless of the direction of head turn. Our findings support previous studies indicating a preference for positive expressions on attractiveness judgements, which is further enhanced by the front views of faces, whether presented during a head turn or shown statically. In sum, our findings imply a complex interaction between cues of social attention, indicated by the view of the face shown, and reward on attractiveness judgements of unfamiliar faces.

Eye Contact and Fear of Being Laughed at in a Gaze Discrimination Task

Current approaches conceptualize gelotophobia as a personality trait characterized by a disproportionate fear of being laughed at by others. Consistently with this perspective, gelotophobes are also described as neurotic and introverted and as having a paranoid tendency to anticipate derision and mockery situations. Although research on gelotophobia has significantly progressed over the past two decades, no evidence exists concerning the potential effects of gelotophobia in reaction to eye contact. Previous research has pointed to difficulties in discriminating gaze direction as the basis of possible misinterpretations of others' intentions or mental states. The aim of the present research was to examine whether gelotophobia predisposition modulates the effects of eye contact (i.e., gaze discrimination) when processing faces portraying several emotional expressions. In two different experiments, participants performed an experimental gaze discrimination task in which they responded, as quickly and accurately as possible, to the eyes' directions on faces displaying either a happy, angry, fear, neutral, or sad emotional expression. In particular, we expected trait-gelotophobia to modulate the eye contact effect, showing specific group differences in the happiness condition. The results of Study 1 (N = 40) indicated that gelotophobes made more errors than non-gelotophobes did in the gaze discrimination task. In contrast to our initial hypothesis, the happiness expression did not have any special role in the observed differences between individuals with high vs. low trait-gelotophobia. In Study 2 (N = 40), we replicated the pattern of data concerning gaze discrimination ability, even after controlling for individuals' scores on social anxiety. Furthermore, in our second experiment, we found that gelotophobes did not exhibit any problem with identifying others' emotions, or a general incorrect attribution of affective features, such as valence, intensity, or arousal. Therefore, this bias in processing gaze might be related to the global processes of social cognition. Further research is needed to explore how eye contact relates to the fear of being laughed at.

Trait Anxiety Impacts the Perceived Gaze Direction of Fearful But Not Angry Faces

Facial expression and gaze direction play an important role in social communication. Previous research has demonstrated the perception of anger is enhanced by direct gaze, whereas, it is unclear whether perception of fear is enhanced by averted gaze. In addition, previous research has shown the anxiety affects the processing of facial expression and gaze direction, but hasn't measured or controlled for depression. As a result, firm conclusions cannot be made regarding the impact of individual differences in anxiety and depression on perceptions of face expressions and gaze direction. The current study attempted to reexamine the effect of the anxiety level on the processing of facial expressions and gaze direction by matching participants on depression scores. A reliable psychophysical index of the range of eye gaze angles judged as being directed at oneself [the cone of direct gaze (CoDG)] was used as the dependent variable in this study. Participants were stratified into high/low trait anxiety groups and asked to judge the gaze of angry, fearful, and neutral faces across a range of gaze directions. The result showed: (1) the perception of gaze direction was influenced by facial expression and this was modulated by trait anxiety. For the high trait anxiety group, the CoDG for angry expressions was wider than for fearful and neutral expressions, and no significant difference emerged between fearful and neutral expressions; For the low trait anxiety group, the CoDG for both angry and fearful expressions was wider than for neutral, and no significant difference emerged between angry and fearful expressions. (2) Trait anxiety modulated the perception of gaze direction only in the fearful condition, such that the fearful CoDG for the high trait anxiety group was narrower than the low trait anxiety group. This demonstrated that anxiety distinctly affected gaze perception in expressions that convey threat (angry, fearful), such that a high trait anxiety level modulated the impact of indirectly threatening expressions (fearful), and did not influence responses to directly threatening expression (angry). These findings partially support the shared signal hypothesis.

Investigating the Effect of Gaze Cues and Emotional Expressions on the Affective Evaluations of Unfamiliar Faces

People look at what they are interested in, and their emotional expressions tend to indicate how they feel about the objects at which they look. The combination of gaze direction and emotional expression can therefore convey important information about people’s evaluations of the objects in their environment, and can even influence the subsequent evaluations of those objects by a third party, a phenomenon known as the emotional gaze effect. The present study extended research into the effect of emotional gaze cues by investigating whether they affect evaluations of the most important aspect of our social environment–other people–and whether the presence of multiple gaze cues enhances this effect. Over four experiments, a factorial within-subjects design employing both null hypothesis significance testing and a Bayesian statistical analysis replicated previous work showing an emotional gaze effect for objects, but found strong evidence that emotional gaze cues do not affect evaluations of other people, and that multiple, simultaneously presented gaze cues do not enhance the emotional gaze effect for either the evaluations of objects or of people. Overall, our results suggest that emotional gaze cues have a relatively weak influence on affective evaluations, especially of those aspects of our environment that automatically elicit affectively valenced reactions, including other humans.

Smiling makes you look older

People smile in social interactions to convey different types of nonverbal communication. However, smiling can potentially change the way a person is perceived along different facial dimensions, including perceived age. It is commonly assumed that smiling faces are perceived as younger than faces carrying a neutral expression. In the series of experiments reported here, I describe an unintuitive and robust effect in the opposite direction. Across different experimental conditions and stimulus sets, smiling faces were consistently perceived as older compared to neutral face photos of the same persons. I suggest that this effect is due to observer failure to ignore smile-associated wrinkles, mainly along the region of the eyes. These findings point to a misconception regarding the relationship between facial smile and perceived age and shed new light on the processes underlying human age perception.

Absence of Sex-Contingent Gaze Direction Aftereffects Suggests a Limit to Contingencies in Face Aftereffects

Face aftereffects (e.g., expression aftereffects) can be simultaneously induced in opposite directions for different face categories (e.g., male and female faces). Such aftereffects are typically interpreted as indicating that distinct neural populations code the categories on which adaptation is contingent, e.g., male and female faces. Moreover, they suggest that these distinct populations selectively respond to variations in the secondary stimulus dimension, e.g., emotional expression. However, contingent aftereffects have now been reported for so many different combinations of face characteristics, that one might question this interpretation. Instead, the selectivity might be generated during the adaptation procedure, for instance as a result of associative learning, and not indicate pre-existing response selectivity in the face perception system. To alleviate this concern, one would need to demonstrate some limit to contingent aftereffects. Here, we report a clear limit, showing that gaze direction aftereffects are not contingent on face sex. We tested 36 young Caucasian adults in a gaze adaptation paradigm. We initially established their ability to discriminate the gaze direction of male and female test faces in a pre-adaptation phase. Afterwards, half of the participants adapted to female faces looking left and male faces looking right, and half adapted to the reverse pairing. We established the effects of this adaptation on the perception of gaze direction in subsequently presented male and female test faces. We found that adaptation induced pronounced gaze direction aftereffects, i.e., participants were biased to perceive small gaze deviations to both the left and right as direct. Importantly, however, aftereffects were identical for male and female test faces, showing that the contingency of face sex and gaze direction participants experienced during the adaptation procedure had no effect.

The Mona Lisa effect: neural correlates of centered and off-centered gaze

The Mona Lisa effect describes the phenomenon when the eyes of a portrait appear to look at the observer regardless of the observer's position. Recently, the metaphor of a cone of gaze has been proposed to describe the range of gaze directions within which a person feels looked at. The width of the gaze cone is about five degrees of visual angle to either side of a given gaze direction. We used functional magnetic resonance imaging to investigate how the brain regions involved in gaze direction discrimination would differ between centered and decentered presentation positions of a portrait exhibiting eye contact. Subjects observed a given portrait's eyes. By presenting portraits with varying gaze directions-eye contact (0°), gaze at the edge of the gaze cone (5°), and clearly averted gaze (10°), we revealed that brain response to gaze at the edge of the gaze cone was similar to that produced by eye contact and different from that produced by averted gaze. Right fusiform gyrus and right superior temporal sulcus showed stronger activation when the gaze was averted as compared to eye contact. Gaze sensitive areas, however, were not affected by the portrait's presentation location. In sum, although the brain clearly distinguishes averted from centered gaze, a substantial change of vantage point does not alter neural activity, thus providing a possible explanation why the feeling of eye contact is upheld even in decentered stimulus positions.

On the facilitative effects of face motion on face recognition and its development

For the past century, researchers have extensively studied human face processing and its development. These studies have advanced our understanding of not only face processing, but also visual processing in general. However, most of what we know about face processing was investigated using static face images as stimuli. Therefore, an important question arises: to what extent does our understanding of static face processing generalize to face processing in real-life contexts in which faces are mostly moving? The present article addresses this question by examining recent studies on moving face processing to uncover the influence of facial movements on face processing and its development. First, we describe evidence on the facilitative effects of facial movements on face recognition and two related theoretical hypotheses: the supplementary information hypothesis and the representation enhancement hypothesis. We then highlight several recent studies suggesting that facial movements optimize face processing by activating specific face processing strategies that accommodate to task requirements. Lastly, we review the influence of facial movements on the development of face processing in the first year of life. We focus on infants' sensitivity to facial movements and explore the facilitative effects of facial movements on infants' face recognition performance. We conclude by outlining several future directions to investigate moving face processing and emphasize the importance of including dynamic aspects of facial information to further understand face processing in real-life contexts.

"Avoiding or approaching eyes"? Introversion/extraversion affects the gaze-cueing effect

We investigated whether the extra-/introversion personality dimension can influence processing of others' eye gaze direction and emotional facial expression during a target detection task. On the basis of previous evidence showing that self-reported trait anxiety can affect gaze-cueing with emotional faces, we also verified whether trait anxiety can modulate the influence of intro-/extraversion on behavioral performance. Fearful, happy, angry or neutral faces, with either direct or averted gaze, were presented before the target appeared in spatial locations congruent or incongruent with stimuli's eye gaze direction. Results showed a significant influence of intra-/extraversion dimension on gaze-cueing effect for angry, happy, and neutral faces with averted gaze. Introverts did not show the gaze congruency effect when viewing angry expressions, but did so with happy and neutral faces; extraverts showed the opposite pattern. Importantly, the influence of intro-/extraversion on gaze-cueing was not mediated by trait anxiety. These findings demonstrated that personality differences can shape processing of interactions between relevant social signals.

The role of facial expressions in attention-orienting in adults and infants

Faces convey many signals (i.e., gaze or expressions) essential for interpersonal interaction. We have previously shown that facial expressions of emotion and gaze direction are processed and integrated in specific combinations early in life. These findings open a number of developmental questions and specifically in this paper we address whether such emotional signals may modulate the behavior in a following gaze context. A classic spatial cueing paradigm was used to assess whether different facial expressions may cause differential orienting response times and modulate the visual response to a peripheral target in adults and in 4-month-old infants. Results showed that both adults and infants oriented towards a peripheral target when a central face was gazing in the direction of the target location. However, in adults this effect occurred regardless of the facial expression displayed by the face. In contrast, in infants, the emotional facial expressions used, at least in the current study, did not facilitate the attention shift but tended to hold infants’ attention.

The Role of Gaze in the Processing of Emotional Facial Expressions

Gaze plays a fundamental role in the processing of facial expressions from birth. Gaze direction is a crucial part of the social signal encoded in and decoded from faces. The ability to discriminate gaze direction, already evident early in life, is essential for the development of more complex socially relevant tasks, such as joint and shared attention. At the same time, facial expressions play a fundamental role in the encoding of gaze direction and, when combined, expression and gaze communicate behavioural motivation to approach or avoid. However, the investigation of how gaze direction and emotional expression interact during the processing of a face has been relatively neglected, and is the key question of this review.

Neurocognitive mechanisms of gaze-expression interactions in face processing and social attention

The face conveys a rich source of non-verbal information used during social communication. While research has revealed how specific facial channels such as emotional expression are processed, little is known about the prioritization and integration of multiple cues in the face during dyadic exchanges. Classic models of face perception have emphasized the segregation of dynamic vs. static facial features along independent information processing pathways. Here we review recent behavioral and neuroscientific evidence suggesting that within the dynamic stream, concurrent changes in eye gaze and emotional expression can yield early independent effects on face judgments and covert shifts of visuospatial attention. These effects are partially segregated within initial visual afferent processing volleys, but are subsequently integrated in limbic regions such as the amygdala or via reentrant visual processing volleys. This spatiotemporal pattern may help to resolve otherwise perplexing discrepancies across behavioral studies of emotional influences on gaze-directed attentional cueing. Theoretical explanations of gaze-expression interactions are discussed, with special consideration of speed-of-processing (discriminability) and contextual (ambiguity) accounts. Future research in this area promises to reveal the mental chronometry of face processing and interpersonal attention, with implications for understanding how social referencing develops in infancy and is impaired in autism and other disorders of social cognition.

Perceptual integrality of componential and configural information in faces

The relative contribution of componential and configural information to face perception is controversial. We addressed this issue in the present study by examining how componential information and configural information interact during face processing, using Garner's (1974) speeded classification paradigm. When classifying upright faces varying in components (eyes, nose, and mouth) and configural information (intereyes and nose-mouth spacing), observers could not selectively attend to components without being influenced by irrelevant variation in configural information, and vice versa, indicating that componential information and configural information are integral in upright face processing. Performance with inverted faces showed selective attention to components but not to configural information, implying dominance of componential information in processing inverted faces. When faces varied only in components, selective attention to different components was observed in upright and inverted faces, indicating that facial components are perceptually separable. These results provide strong evidence that integrality of componential and configural information, rather than the relative dominance of either, is the hallmark of upright face perception.

The Interaction Between Gaze and Facial Expression in the Amygdala and Extended Amygdala is Modulated by Anxiety

Behavioral evidence indicates that angry faces are seen as more threatening, and elicit greater anxiety, when directed at the observer, whereas the influence of gaze on the processing of fearful faces is less consistent. Recent research has also found inconsistent effects of expression and gaze direction on the amygdala response to facial signals of threat. However, such studies have failed to consider the important influence of anxiety on the response to signals of threat; an influence that is well established in behavioral research and recent neuroimaging studies. Here, we investigated the way in which individual differences in anxiety would influence the interactive effect of gaze and expression on the response to angry and fearful faces in the human extended amygdala. Participants viewed images of fearful, angry and neutral faces, either displaying an averted or direct gaze. We found that state anxiety predicted an increased response in the dorsal amygdala/substantia innominata (SI) to angry faces when gazing at, relative to away from the observer. By contrast, high state anxious individuals showed an increased amygdala response to fearful faces that was less dependent on gaze. In addition, the relationship between state anxiety and gaze on emotional intensity ratings mirrored the relationship between anxiety and the amygdala/SI response. These results have implications for understanding the functional role of the amygdala and extended amygdala in processing signals of threat, and are consistent with the proposed role of this region in coding the relevance or significance of a stimulus to the observer.

Eye gaze during face processing in children and adolescents with 22q11.2 deletion syndrome

OBJECTIVE

The 22q11.2 deletion syndrome (22q11DS) is a neurogenetic syndrome with high risk for the development of psychiatric disorder. There is interest in identifying reliable markers for measuring and monitoring socio-emotional impairments in 22q11DS during development. The current study investigated eye gaze as a potential marker during a face-processing task in children and young adolescents with 22q11DS.

METHOD

Eye gaze and behavioral correlates were investigated in 26 subjects (aged 8 to 15 years) with 22q11DS during the Jane Task, which targets featural and configural face processing. Individuals with 22q11DS were compared with chronologically age-matched healthy controls and individuals with idiopathic developmental delay (DD).

RESULTS

Few differences in accuracy were observed between patients with 22q11DS and DD controls; however individuals with 22q11DS spent less time on the eyes and more time on the mouths than both comparison groups. IQ predicted time on the eyes in subjects with 22q11DS, and anxiety predicted time on the eyes in DD and 22q11DS subjects.

CONCLUSIONS

These results provide evidence for abnormal exploration of faces in the syndrome and suggest that time spent on the eyes may contribute to face processing difficulties and interact with anxiety levels to exacerbate socio-emotional dysfunction in affected individuals.

The effect of face eccentricity on the perception of gaze direction

The perception of a looker's gaze direction depends not only on iris eccentricity (the position of the looker's irises within the sclera) but also on the orientation of the lookers' head. One among several potential cues of head orientation is face eccentricity, the position of the inner features of the face (eyes, nose, mouth) within the head contour, as viewed by the observer. For natural faces this cue is confounded with many other head-orientation cues, but in schematic faces it can be studied in isolation. Salient novel illustrations of the effectiveness of face eccentricity are 'Necker faces', which involve equal iris eccentricities but multiple perceived gaze directions. In four experiments, iris and face eccentricity in schematic faces were manipulated, revealing strong and consistent effects of face eccentricity on perceived gaze direction, with different types of tasks. An additional experiment confirmed the 'Mona Lisa' effect with this type of stimuli. Face eccentricity most likely acted as a simple but robust cue of head turn. A simple computational account of combined effects of cues of eye and head turn on perceived gaze direction is presented, including a formal condition for the perception of direct gaze. An account of the 'Mona Lisa' effect is presented.

Does gaze direction modulate facial expression processing in children with autism spectrum disorder?

Two experiments investigated whether children with autism spectrum disorder (ASD) integrate relevant communicative signals, such as gaze direction, when decoding a facial expression. In Experiment 1, typically developing children (9-14 years old; n = 14) were faster at detecting a facial expression accompanying a gaze direction with a congruent motivational tendency (i.e., an avoidant facial expression with averted eye gaze) than those with an incongruent motivational tendency. Children with ASD (9-14 years old; n = 14) were not affected by the gaze direction of facial stimuli. This finding was replicated in Experiment 2, which presented only the eye region of the face to typically developing children (n = 10) and children with ASD (n = 10). These results demonstrated that children with ASD do not encode and/or integrate multiple communicative signals based on their affective or motivational tendency.

Event-related potentials reveal temporal staging of dynamic facial expression and gaze shift effects on attentional orienting

Multiple sources of information from the face guide attention during social interaction. The present study modified the Posner cueing paradigm to investigate how dynamic changes in emotional expression and eye gaze in faces affect the neural processing of subsequent target stimuli. Event-related potentials (ERPs) were recorded while participants viewed centrally presented face displays in which gaze direction (left, direct, right) and facial expression (fearful, neutral) covaried in a fully crossed design. Gaze direction was not predictive of peripheral target location. ERP analysis revealed several sequential effects, including: (1) an early enhancement of target processing following fearful faces (P1); (2) an interaction between expression and gaze (N1), with enhanced target processing following fearful faces with rightward gaze; and (3) an interaction between gaze and target location (P3), with enhanced processing for invalidly cued left visual field targets. Behaviorally, participants responded faster to targets following fearful faces and targets presented in the right visual field, in concordance with the P1 and N1 effects, respectively. The findings indicate that two nonverbal social cues-facial expression and gaze direction-modulate attentional orienting across different temporal stages of processing. Results have implications for understanding the mental chronometry of shared attention and social referencing.

Interactive processes in matching identity and expressions of unfamiliar faces: evidence for mutual facilitation effects

We investigated the interactions between matching identity and expressions of unfamiliar faces. In experiment 1, participants matched expressions in frontal and in oblique views, while we manipulated facial identity. In experiment 2, participants matched identity in frontal and in oblique views, while facial expressions were manipulated. Labeling of expressions was not required. Results showed mutual facilitation between matching facial identity and facial expressions, in accuracy as well as in reaction times. Thus, matching expressions was better and faster for same-identity images in oblique as well as in frontal views (experiment 1), and matching identity was better and faster for same-expression images in oblique as well as in frontal views (experiment 2). The discussion focuses on the implications of these results for the structural encoding of facial identity and facial expressions.

Pointing with the eyes: the role of gaze in communicating danger

Facial expression and direction of gaze are two important sources of social information, and what message each conveys may ultimately depend on how the respective information interacts in the eye of the perceiver. Direct gaze signals an interaction with the observer but averted gaze amounts to "pointing with the eyes", and in combination with a fearful facial expression may signal the presence of environmental danger. We used fMRI to examine how gaze direction influences brain processing of facial expression of fear. The combination of fearful faces and averted gazes activated areas related to gaze shifting (STS, IPS) and fear-processing (amygdala, hypothalamus, pallidum). Additional modulation of activation was observed in motion detection areas, in premotor areas and in the somatosensory cortex, bilaterally. Our results indicate that the direction of gaze prompts a process whereby the brain combines the meaning of the facial expression with the information provided by gaze direction, and in the process computes the behavioral implications for the observer.

A role for eyebrows in regulating the visibility of eye gaze direction

The human eye is unique amongst those of primates in having white sclera against which the dark iris is clearly visible. This high-contrast structure makes the gaze direction of a human potentially easily perceptible to others. For a social creature such as a human, the ability to perceive the direction of another's gaze may be very useful, since gaze usually signals attention. We report data showing that the accuracy of gaze deviation detection is independent of viewing distance up to a certain critical distance, beyond which it collapses. This is, of itself, surprising since most visual tasks are performed better at closer viewing distances. Our data also show that the critical distance, but not accuracy, is affected by the position of the eyebrows so that lowering the eyebrows reduces the critical distance. These findings show that mechanisms exist by which humans could expand or restrict the availability of their gaze direction to others. A way to regulate the availability of the gaze direction signal could be an advantage. We show that an interpretation of eyebrow function in these terms provides a novel explanation for several well-known eyebrow actions, including the eyebrow flash.

The effects of eye and face inversion on the early stages of gaze direction perception--an ERP study

Eye direction perception is highly important for social cognition. However, the neural mechanism underlying gaze direction perception has not been well elucidated. The present study aimed to examine the specific neural mechanism of gaze direction perception by investigating how the event related potential components, which presumably reflect the early stages of face processing, are affected by inverting eye region and face context, i.e., facial parts other than eye region. The results showed that eye region inversion significantly delayed the peak latency of the N170 component. At the same time, N170 latency was also delayed by inverting face context alone. Moreover, we observed that the P100 latency was delayed by inverting the eye region in an upright face context, but not in an inverted face context. We suggest that N170 reflects the eye-sensitive cortical response, but also the processing of other facial regions, and that the processing of eye region begins at an early stage of face processing, around 100 ms after stimulus onset.

Facial expression and gaze-direction in human superior temporal sulcus

The perception of facial expression and gaze-direction are important aspects of non-verbal communication. Expressions communicate the internal emotional state of others while gaze-direction offers clues to their attentional focus and future intentions. Cortical regions in the superior temporal sulcus (STS) play a central role in the perception of expression and gaze, but the extent to which the neural representations of these facial gestures are overlapping is unknown. In the current study 12 subjects observed neutral faces with direct-gaze, neutral faces with averted-gaze, or emotionally expressive faces with direct-gaze while we scanned their brains with functional magnetic resonance imaging (fMRI), allowing a comparison of the hemodynamic responses evoked by perception of expression and averted-gaze. The inferior occipital gyri, fusiform gyri, STS and inferior frontal gyrus were more strongly activated when subjects saw facial expressions than when they saw neutral faces. The right STS was more strongly activated by the perception of averted-gaze than direct-gaze faces. A comparison of the responses within right STS revealed that expression and averted-gaze activated distinct, though overlapping, regions of cortex. We propose that gaze-direction and expression are represented by dissociable overlapping neural systems.

The recognition of emotional expression in prosopagnosia: decoding whole and part faces

Prosopagnosia is currently viewed within the constraints of two competing theories of face recognition, one highlighting the analysis of features, the other focusing on configural processing of the whole face. This study investigated the role of feature analysis versus whole face configural processing in the recognition of facial expression. A prosopagnosic patient, SC made expression decisions from whole and incomplete (eyes-only and mouth-only) faces where features had been obscured. SC was impaired at recognizing some (e.g., anger, sadness, and fear), but not all (e.g., happiness) emotional expressions from the whole face. Analyses of his performance on incomplete faces indicated that his recognition of some expressions actually improved relative to his performance on the whole face condition. We argue that in SC interference from damaged configural processes seem to override an intact ability to utilize part-based or local feature cues.

Geometrical basis of perception of gaze direction

Perception of gaze direction depends not only on the position of the irises within the looker's eyes but also on the orientation of the looker's head. A simple analysis of the geometry of gaze direction predicts this dependence. This analysis is applied to explain the Wollaston effect, the Mona Lisa effect, and the newly presented Mirror gaze effect. In an experiment synthetic faces were used in which the position of the iris and the angle of head rotation were varied. Different groups of subjects judged iris position, head rotation, and gaze direction of the same stimuli. The results illustrate how cues of iris location and head orientation interact to determine perceived gaze direction.

Eye contact influences neural processing of emotional expressions in 4-month-old infants

Eye gaze is a fundamental component of human communication. During the first post-natal year, infants rapidly learn that the gaze of others provides socially significant information. In addition, infants are sensitive to several emotional expressions. However, little is known regarding how eye contact influences the way the infant brain processes emotional expressions. We measured 4-month-old infants' brain electric activity to assess neural processing of faces displaying neutral, happy and angry emotional expressions when accompanied by direct and averted eye gaze. The results show that processing of angry facial expressions was influenced by eye gaze. In particular, infants showed enhanced neural processing of angry expressions when these expressions were accompanied by direct eye gaze. These results show that by 4 months of age, the infant detects angry emotional expressions, and the infant brain processes their relevance to the self.

Do the upright eyes have it?

Eye contact is crucial for social communication. A perceived direct gaze facilitates detection, whereas face inversion diminishes this facilitative effect (Senju, Hasegawa, & Tojo, 2005). In the present study, we adopted a visual search paradigm to investigate why a direct gaze facilitates detection in an upright face, but not in an upside-down face. Upright eyes were found to facilitate detection even when other parts of the face were inverted or absent, whereas inverted eyes had no effect on search performance. A critical role for the morphological information of upright eyes, which can be distorted by "eye inversion," in direct gaze processing is suggested.