Eyes always attract attention but gaze orienting is task-dependent: Evidence from eye movement monitoring

How the brain differentiates human and monkey faces: Neuromagnetic evidence from spatiotemporal dynamics

Face recognition is an important aspect of human non-verbal communication. Event-related potentials or magnetic fields, such as the N170/M170 component, are considered essential neural markers of face processing. Compared to upright human faces, inverted human faces and upright but not inverted animal faces cause longer latencies and larger amplitudes of these components. However, the mechanisms underlying this factor remain unclear. To elucidate the spatiotemporal dynamics of the processing of inverted human and animal faces, we recorded face-selective responses (M170) to upright and inverted human and monkey faces using a 306-channel whole-head magnetoencephalography. Sensor analysis showed an increased M170 latency and amplitude for inverted human and upright animal faces. However, in the source analysis, the observed modulations of the estimated spatiotemporal dynamics were different from the sensor results: irrespective of species, upright faces activated wider areas in the ventral and dorsal visual regions compared with inverted faces. Additionally, face orientation differentially modulated the anterior region of the fusiform gyrus (FG) in both face categories. These results suggest that spatiotemporal dynamics differ in face orientation regardless of category and that the FG contributes little or nothing to the M170 modulation recorded in the scalp sensor. Furthermore, we demonstrated that inverted human and animal faces are processed via different mechanisms.

Social attention in the wild - fixations to the eyes and autistic traits during a naturalistic interaction in a healthy sample

Attention to social stimuli is a key component of social behavior and facilitates the development of fundamental social skills. Studies investigating social attention in neurotypical or neurodiverse populations have often relied on screen-based experiments using static images or videos, which lack the sensory richness and reciprocity present in real-life social interactions. This can possibly be attributed to the challenges one encounters when creating naturalistic experiments, such as dealing with dynamically moving areas of interest (AOIs), which require either time-intensive manual coding or restraining of participants. Here, we present findings from an experimental paradigm using unrestrained mobile eye-tracking and a face detection algorithm (MTCNN) to measure fixation rates during a semi-structured, face-to-face interview. Data from N = 62 healthy adult participants was analyzed for gaze behavior and related to participants' autistic traits. We observed a significant negative correlation between fixation rates on the eye region averaged over the entire interaction and scores on the autism spectrum quotient (AQ) (r = -0.14), indicating participants with high autistic traits fixated less frequently on the eye region. We also compared different types of interview questions (open vs. closed) to explore whether the reduction in fixation rates was more pronounced for specific time intervals during the interview. Lastly, we discuss both possibilities for extensions as well as limitations of the presented paradigm that could serve as inspiration for future research.

Effects of emotional content on social inhibition of gaze in live social and non-social situations

In real-life interactions, it is crucial that humans adequately respond to others' emotional expressions. Emotion perception so far has mainly been studied in highly controlled laboratory tasks. However, recent research suggests that attention and gaze behaviour significantly differ between watching a person on a controlled laboratory screen compared to in real world interactions. Therefore, the current study aimed to investigate effects of emotional expression on participants' gaze in social and non-social situations. We compared looking behaviour towards a confederate showing positive, neutral or negative facial expressions between live social and non-social waiting room situations. Participants looked more often and longer to the confederate on the screen, than when physically present in the room. Expressions displayed by the confederate and individual traits (social anxiety and autistic traits) of participants did not reliably relate to gaze behaviour. Indications of covert attention also occurred more often and longer during the non-social, than during the social condition. Findings indicate that social norm is a strong factor modulating gaze behaviour in social contexts. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on September 13, 2021. The protocol, as accepted by the journal, can be found at: https://doi.org/10.6084/m9.figshare.16628290 .

Gaze and speech behavior in parent–child interactions: The role of conflict and cooperation

A primary mode of human social behavior is face-to-face interaction. In this study, we investigated the characteristics of gaze and its relation to speech behavior during video-mediated face-to-face interactions between parents and their preadolescent children. 81 parent–child dyads engaged in conversations about cooperative and conflictive family topics. We used a dual-eye tracking setup that is capable of concurrently recording eye movements, frontal video, and audio from two conversational partners. Our results show that children spoke more in the cooperation-scenario whereas parents spoke more in the conflict-scenario. Parents gazed slightly more at the eyes of their children in the conflict-scenario compared to the cooperation-scenario. Both parents and children looked more at the other's mouth region while listening compared to while speaking. Results are discussed in terms of the role that parents and children take during cooperative and conflictive interactions and how gaze behavior may support and coordinate such interactions.

How We Perceive Others Resembling Us

Eye contact is essential for social cognition, acting as an important tool for social communication. While differences in face scanning patterns concerning familiarity have been thoroughly investigated, the impact of facial similarity on gaze behavior has not been examined yet. We addressed this topic by recording subjects' eye-directed gazing while looking at faces that were individually created systematically varying in terms of similarity to the self-face and familiarity. Subjects' self-faces were morphed into three other faces including a close friend of the same sex. Afterwards, they rated similarity to their self-face of those morphed face stimuli in a separate rating task. Our results show a general preference for the eyes' area as well as differences regarding fixation patterns depending on similarity to the self-face. The lower the similarity to the self-face, the more fixations on the eyes' area. Subjects' ratings followed a linear line, indicating well-pronounced face perception. Nevertheless, other faces were rated faster than the self-face independent of familiarity, while morphed faces got the slowest ratings. Our results mirror the importance of similarity to the self-face as a factor shaping the way we look at the eyes of others explaining variance apart from familiarity.

How does gaze to faces support face-to-face interaction? A review and perspective

Gaze-where one looks, how long, and when-plays an essential part in human social behavior. While many aspects of social gaze have been reviewed, there is no comprehensive review or theoretical framework that describes how gaze to faces supports face-to-face interaction. In this review, I address the following questions: (1) When does gaze need to be allocated to a particular region of a face in order to provide the relevant information for successful interaction; (2) How do humans look at other people, and faces in particular, regardless of whether gaze needs to be directed at a particular region to acquire the relevant visual information; (3) How does gaze support the regulation of interaction? The work reviewed spans psychophysical research, observational research, and eye-tracking research in both lab-based and interactive contexts. Based on the literature overview, I sketch a framework for future research based on dynamic systems theory. The framework holds that gaze should be investigated in relation to sub-states of the interaction, encompassing sub-states of the interactors, the content of the interaction as well as the interactive context. The relevant sub-states for understanding gaze in interaction vary over different timescales from microgenesis to ontogenesis and phylogenesis. The framework has important implications for vision science, psychopathology, developmental science, and social robotics.

Implying social interaction and its influence on gaze behavior to the eyes

Researchers have increasingly focused on how the potential for social interaction modulates basic processes of visual attention and gaze behavior. In this study, we investigated why people may experience social interaction and what factors contributed to their subjective experience. We furthermore investigated whether implying social interaction modulated gaze behavior to people’s faces, specifically the eyes. To imply the potential for interaction, participants received either one of two instructions: 1) they would be presented with a person via a ‘live’ video-feed, or 2) they would be presented with a pre-recorded video clip of a person. Prior to the presentation, a confederate walked into a separate room to suggest to participants that (s)he was being positioned behind a webcam. In fact, all participants were presented with a pre-recorded clip. During the presentation, we measured participants’ gaze behavior with an eye tracker, and after the presentation, participants were asked whether they believed that the confederate was ‘live’ or not, and, why they thought so. Participants varied greatly in their judgements about whether the confederate was ‘live’ or not. Analyses of gaze behavior revealed that a large subset of participants who received the live-instruction gazed less at the eyes of confederates compared with participants who received the pre-recorded-instruction. However, for both the live-instruction group and the pre-recorded instruction group, another subset of participants gazed predominantly at the eyes. The current findings may contribute to the development of experimental designs aimed to capture the interactive aspects of social cognition and visual attention.

Eastern observers cannot inhibit their gaze to eye and nose regions in face perception

It has been found that Western observers cannot inhibit their gaze to the eye region, even if they are told to avoid doing so when they observe face stimuli because of the importance of the eye region. However, studies indicate that the nose region is more important for face processing among Eastern observers. We used the "don't look" paradigm with Eastern observers, in which participants were told to avoid fixating on a specific region (eye, nose, and mouth). The results extend previous findings as both the eye and nose regions attracted their gaze. Interestingly, the fixation behaviors differed for the eyes and nose in terms of the time-dependent view, in which reflexive saccades to the eye with a persistent fixation to the nose were observed. The nose regions could have stronger attractiveness than previously thought.

Artificial Faces Predict Gaze Allocation in Complex Dynamic Scenes

Both low-level physical saliency and social information, as presented by human heads or bodies, are known to drive gaze behavior in free-viewing tasks. Researchers have previously made use of a great variety of face stimuli, ranging from photographs of real humans to schematic faces, frequently without systematically differentiating between the two. In the current study, we used a Generalized Linear Mixed Model (GLMM) approach to investigate to what extent schematic artificial faces can predict gaze when they are presented alone or in competition with real human faces. Relative differences in predictive power became apparent, while GLMMs suggest substantial effects for real and artificial faces in all conditions. Artificial faces were accordingly less predictive than real human faces but still contributed significantly to gaze allocation. These results help to further our understanding of how social information guides gaze in complex naturalistic scenes.

Facial Reactivity and Attentional Processing of Facial Expressions and Gaze Direction

Abstract. Patterns of facial reactivity and attentional allocation to emotional facial expressions, and how these are moderated by gaze direction, are not clearly established. Among a sample of undergraduate university students, aged between 17 and 22 years (76% female), corrugator and zygomatic reactivity, as measured by facial electromyography, and attention allocation, as measured by the startle reflex and startle-elicited N100, was examined while viewing happy, neutral, angry and fearful facial expressions, which were presented at either 0- or 30-degree gaze. Results indicated typically observed facial mimicry to happy faces but, unexpectedly, “smiling” facial responses to fearful, and to a lesser extent, angry faces. This facial reactivity was not influenced by gaze direction. Furthermore, emotional facial expressions did not elicit increased attentional allocation. Likewise, matched facial expressions did not elicit increased attentional allocation. Rather, happy and fearful faces with direct (0°) gaze elicited increased controlled attentional allocation, and averted (30°) gaze faces, regardless of emotional expression, elicited preferential, early cortical processing. These findings suggest typical facial mimicry to happy faces, but unexpected facial reactivity to angry and fearful faces, perhaps due to an attempt to regulate social bonds during threat perception. Findings also suggest a divergence in controlled versus preferential, early cortical attentional processing for direct compared to averted gaze faces. These findings relate to young, mostly female, adults attending university. The experiment should be repeated with a larger sample drawn from the general community, with a broader age range and gender balance, and with a stimulus set with validated subjective valence and arousal ratings. This can reduce Type II error and establish normative patterns of facial reactivity and attentional processing of emotional facial expressions with different gaze directions.

Socially Communicative Eye Contact and Gender Affect Memory

Because of their value as a socially communicative cue, researchers have strived to understand how the gaze of other people influences a variety of cognitive processes. Recent work in social attention suggests that the use of images of people in laboratory studies, as a substitute for real people, may not effectively test socially communicative aspects of eye gaze. As attention affects many other cognitive processes, it is likely that social attention between real individuals could also affect other cognitive processes, such as memory. However, from previous work alone, it is unclear whether, and if so how, socially communicative eye gaze affects memory. The present studies test the assumption that socially communicative aspects of eye gaze may impact memory by manipulating the eye gaze of a live speaker in the context of a traditional recognition paradigm used frequently in the laboratory. A female (Experiment 1) or male (Experiment 2) investigator read words aloud and varied whether eye contact was, or was not, made with a participant. With both female and male investigators, eye contact improved word recognition only for female participants and hindered word recognition in male participants. When a female investigator prolonged their eye contact (Experiment 3) to provide a longer opportunity to both observe and process the investigator's eye gaze, the results replicated the findings from Experiments 1 and 2. The findings from Experiments 1-3 suggest that females interpret and use the investigator's eye gaze differently than males. When key aspects from the previous experiments were replicated in a noncommunicative situation (i.e., when a video of a speaker is used instead of a live speaker; Experiment 4), the memory effects observed previously in response to eye gaze were eliminated. Together, these studies suggest that it is the socially communicative aspects of eye gaze from a real person that influence memory. The findings reveal the importance of using social cues that are communicative in nature (e.g., real people) when studying the relationship between social attention and memory.

Differential impact of trait, social, and attachment anxiety on the stare-in-the-crowd effect

Eye gaze conveys crucial information for social interactions, with straight versus averted gaze triggering distinct emotional and cognitive processes. The "stare-in-the-crowd" effect exemplifies such differential visual processing of gaze direction, in more recent reports also in interaction with head orientation. Besides aiming at replicating the "stare-in-the-crowd" effect by means of an eye gaze by head orientation interaction, the present study intended to for the first time testing its susceptibility to inter-individual differences in trait, social, and attachment anxiety. Our findings reveal a significant relation between the "stare-in-the-crowd" effect and social and attachment, but not trait anxiety, and therefore provide preliminary cues for personality influences on visual processing of eye gaze and head orientation.

Why are you looking at me? It's because I'm talking, but mostly because I'm staring or not doing much

Our attention is particularly driven toward faces, especially the eyes, and there is much debate over the factors that modulate this social attentional orienting. Most of the previous research has presented faces in isolation, and we tried to address this shortcoming by measuring people's eye movements whilst they observe more naturalistic and varied social interactions. Participants' eye movements were monitored whilst they watched three different types of social interactions (monologue, manual activity, active attentional misdirection), which were either accompanied by the corresponding audio as speech or by silence. Our results showed that (1) participants spent more time looking at the face when the person was giving a monologue, than when he/she was carrying out manual activities, and in the latter case they spent more time fixating on the person's hands. (2) Hearing speech significantly increases the amount of time participants spent looking at the face (this effect was relatively small), although this was not accounted for by any increase in mouth-oriented gaze. (3) Participants spent significantly more time fixating on the face when direct eye contact was established, and this drive to establish eye contact was significantly stronger in the manual activities than during the monologue. These results highlight people's strategic top-down control over when they attend to faces and the eyes, and support the view that we use our eyes to signal non-verbal information.

Early and late cortical responses to directly gazing faces are task dependent

Gender categorisation of human faces is facilitated when gaze is directed toward the observer (i.e., a direct gaze), compared with situations where gaze is averted or the eyes are closed (Macrae, Hood, Milne, Rowe, & Mason, Psychological Science, 13(5), 460-464, 2002). However, the temporal dynamics underlying this phenomenon remain to some extent unknown. Here, we used electroencephalography (EEG) to assess the neural correlates of this effect, focusing on the event-related potential (ERP) components known to be sensitive to gaze perception (i.e., P1, N170, and P3b). We first replicated the seminal findings of Macrae et al. (2002, Experiment 1) regarding facilitated gender discrimination, and subsequently measured the underlying neural responses. Our data revealed an early preferential processing of direct gaze as compared with averted gaze and closed eyes at the P1, which reverberated at the P3b (Experiment 2). Critically, using the same material, we failed to reproduce these effects when gender categorisation was not required (Experiment 3). Taken together, our data confirm that direct gaze enhances the early P1, as well as later cortical responses to face processing, although the effect appears to be task dependent.

Perceived Gaze Direction Modulates Neural Processing of Prosocial Decision Making

Gaze direction is a common social cue implying potential interpersonal interaction. However, little is known about the neural processing of social decision making influenced by perceived gaze direction. Here, we employed functional magnetic resonance imaging (fMRI) method to investigate 27 females when they were engaging in an economic exchange game task during which photos of direct or averted eye gaze were shown. We found that, when averted but not direct gaze was presented, prosocial vs. selfish choices were associated with stronger activations in the right superior temporal gyrus (STG) as well as larger functional couplings between right STG and the posterior cingulate cortex (PCC). Moreover, stronger activations in right STG was associated with quicker actions for making prosocial choice accompanied with averted gaze. The findings suggest that, when the cue implying social contact is absent, the processing of understanding others’ intention and the relationship between self and others is more involved for making prosocial than selfish decisions. These findings could advance our understanding of the roles of subtle cues in influencing prosocial decision making, as well as shedding lights on deficient social cue processing and functioning among individuals with autism spectrum disorder (ASD).

Spatial attention modulates the temporal sequence of hemispheric asymmetry in configural and featural face processing

Face recognition requires both configural and featural processing. Configural face processing is more dependent on the right hemisphere, whereas featural face processing is more dependent on the left hemisphere. The ERP components sensitive to configural and featural face processing were found on P1 and P2, respectively. However, whether lateralized processing is independent of or interacts with the temporal sequence of configural and featural face processing is unclear. To prevent potentially confounding physical stimuli differences between configural and featural face processing from affecting the ERP components, a spatial attention paradigm was employed in which the participants were instructed to attend to the face location (the attended face condition) or the house location (the unattended face condition). The interaction effect of attention, face processing type and hemisphere on the P1 and P2 components indicates that the different mechanisms of configural and featural face processing are a function of spatial attention. Specifically, under the attended face condition, the posterior P1 (approximately 100 ms) for configural face processing was larger than that for featural face processing in the right hemisphere, whereas the P2 (approximately 220 ms) for featural face processing was larger than that for configural face processing in the left hemisphere. In contrast, under the unattended face condition, the P1 for featural face processing was larger than that for configural face processing in the left hemisphere, whereas the P2 for configural face processing was larger than that for featural face processing in the right hemisphere. Therefore, configural and featural processing involve different neural mechanisms, and more importantly, the time course of hemispheric asymmetry in configural and featural face processing is differentially modulated by spatial attention.

Watching More Closely: Shot Scale Affects Film Viewers' Theory of Mind Tendency But Not Ability

Recent research debates the effects of exposure to narrative fiction on recognition of mental states in others and self, referred to as Theory of Mind. The current study explores the mechanisms by which such effects could occur in fictional film. Using manipulated film scenes, we conducted a between subject experiment (N = 136) exploring how film shot-scale affects viewers' Theory of Mind. Specifically, in our methods we distinguish between the trait Theory of Mind abilities (ToM ability), and the state-like tendency to recognize mental states in others and self (ToM tendency). Results showed that close-up shots (compared to long shots) of a character was associated with higher levels of Theory of Mind tendency, when the facial expression was sad but not when it was neutral. And this effect did not transfer to other characters in the film. There was also no observable effect of character depiction on viewers' general Theory of Mind ability. Together the findings suggest that formal and content features of shot scale can elicit Theory of Mind responses by directing attention toward character mental states rather than improving viewers' general Theory of Mind ability.

Own-age biases in adults’ and children’s joint attention: Biased face prioritization, but not gaze following!

Previous studies have reported own-age biases in younger and older adults in gaze following. We investigated own-age biases in social attentional processes between adults and children by focusing on two aspects of the joint attention process; the extent to which people attend towards an individual’s face, and the extent to which they fixate objects that are looked at by this person (i.e., gaze following). Participants viewed images that always contained a child and an adult who either looked towards each other or each looked at objects located to their side. Observers consistently, and rapidly fixated the actor’s faces, though the children were faster to fixate the child’s face than the adult’s faces, whilst the adults were faster to fixate on the adult’s face than the child’s face. The children also spent significantly more time fixating the child’s face than the adult’s face, and the opposite pattern of results was found for the adults. Whilst both adults and children prioritized objects when they were looked at by the actor, both groups showed equivalent levels of gaze following, and there was no own-age bias for gaze following. Our results show an own-age bias for prioritizing faces, but not gaze following.

Comparison of choose-a-movie and approach-avoidance paradigms to measure social motivation

Social motivation is a subjective state which is rather difficult to quantify. It has sometimes been conceptualised as "behavioural effort" to seek social contact. Two paradigms: approach-avoidance (AA) and choose a movie (CAM), based on the same conceptualisation, have been used to measure social motivation in people with and without autism. However, in absence of a direct comparison, it is hard to know which of these paradigms has higher sensitivity in estimating preference for social over non-social stimuli. Here we compare these two tasks for their utility in (1) evaluating social seeking in typical people and (2) identifying the influence of autistic traits on social motivation. Our results suggest that CAM reveals a clear preference for social stimuli over non-social in typical adults but AA fails to do so. Also, social seeking measured with CAM but not AA has a negative relationship between autistic traits.

Are you looking my way? Ostracism widens the cone of gaze

Ostracized individuals demonstrate an increased need for belonging. To satisfy this need, they search for signals of inclusion, one of which may be another person's gaze directed at oneself. We tested if ostracized, compared to included, individuals judge a greater degree of averted gaze as still being direct. This range of gaze angles still viewed as direct has been dubbed “the cone of (direct) gaze”. In the current research, ostracized and included participants viewed friendly-looking face stimuli with direct or slightly averted gaze (0°, 2°, 4°, 6°, and 8° to the left and to the right) and judged whether stimulus persons were looking at them or not. Ostracized individuals demonstrated a wider gaze cone than included individuals.

Orienting towards social features in naturalistic scenes is reflexive

Saliency-based models of visual attention postulate that, when a scene is freely viewed, attention is predominantly allocated to those elements that stand out in terms of their physical properties. However, eye-tracking studies have shown that saliency models fail to predict gaze behavior accurately when social information is included in an image. Notably, gaze pattern analyses revealed that depictions of human beings are heavily prioritized independent of their low-level physical saliency. What remains unknown, however, is whether the prioritization of such social features is a reflexive or a voluntary process. To investigate the early stages of social attention in more detail, participants viewed photographs of naturalistic scenes with and without social features (i.e., human heads or bodies) for 200 ms while their eye movements were being recorded. We observed significantly more first eye movements to regions containing social features than would be expected from a chance level distribution of saccades. Additionally, a generalized linear mixed model analysis revealed that the social content of a region better predicted first saccade direction than its saliency suggesting that social features partially override the impact of low-level physical saliency on gaze patterns. Given the brief image presentation time that precluded visual exploration, our results provide compelling evidence for a reflexive component in social attention. Moreover, the present study emphasizes the importance of considering social influences for a more coherent understanding of human attentional selection.

Genuine eye contact elicits self-referential processing

The effect of eye contact on self-awareness was investigated with implicit measures based on the use of first-person singular pronouns in sentences. The measures were proposed to tap into self-referential processing, that is, information processing associated with self-awareness. In addition, participants filled in a questionnaire measuring explicit self-awareness. In Experiment 1, the stimulus was a video clip showing another person and, in Experiment 2, the stimulus was a live person. In both experiments, participants were divided into two groups and presented with the stimulus person either making eye contact or gazing downward, depending on the group assignment. During the task, the gaze stimulus was presented before each trial of the pronoun-selection task. Eye contact was found to increase the use of first-person pronouns, but only when participants were facing a real person, not when they were looking at a video of a person. No difference in self-reported self-awareness was found between the two gaze direction groups in either experiment. The results indicate that eye contact elicits self-referential processing, but the effect may be stronger, or possibly limited to, live interaction.

Facing the partner influences exchanges in force

Many studies in psychology have documented how the behaviour of verbally communicating pairs is affected by social factors such as the partner’s gaze. However, few studies have examined whether physically interacting pairs are influenced by social factors. Here, we asked two partners to exchange forces with one another, where the goal was to accurately replicate the force back onto the other. We first measured an individual’s accuracy in reproducing a force from a robot. We then tested pairs who knowingly exchanged forces whilst separated by a curtain. These separated pairs exchanged forces as two independent individuals would, hence the force reproduction accuracy of partners is not affected by knowingly reproducing a force onto a nonvisible partner. On the other hand, pairs who exchanged forces whilst facing one another consistently under-reproduced the partner’s force in comparison to separated partners. Thus, the force reproduction accuracy of subjects is strongly biased by facing a partner.

Real Person Interaction in Visual Attention Research

Abstract. An important development in cognitive psychology in the past decade has been the examination of visual attention during real social interaction. This contrasts traditional laboratory studies of attention, including “social attention,” in which observers perform tasks alone. In this review, we show that although the lone-observer method has been central to attention research, real person interaction paradigms have not only uncovered the processes that occur during “joint attention,” but have also revealed attentional processes previously thought not to occur. Furthermore, the examination of some visual attention processes almost invariably requires the use of real person paradigms. While we do not argue for an increase in “ecological validity” for its own sake, we do suggest that research using real person interaction has greatly benefited the development of visual attention theories.

Responding to social and symbolic extrafoveal cues: cue shape trumps biological relevance

Social cues presented at visual fixation have been shown to strongly influence an observer’s attention and response selection. Here we ask whether the same holds for cues (initially) presented away from fixation, as cues are commonly perceived in natural vision. In six experiments, we show that extrafoveally presented cues with a distinct outline, such as pointing hands, rotated heads, and arrow cues result in strong cueing of responses (either to the cue itself, or a cued object). In contrast, cues without a clear outline, such as gazing eyes and direction words exert much weaker effects on participants’ responses to a target cue. We also show that distraction effects on response times are relatively weak, but that strong interference effects can be obtained by measuring mouse trajectories. Eye tracking suggests that gaze cues are slower to respond to because their direction cannot easily be perceived in extrafoveal vision. Together, these data suggest that the strength of an extrafoveal cue is determined by the shape of the cue outline, rather than its biological relevance (i.e., whether the cue is provided by another human being), and that this shape effect is due to how easily the direction of a cue can be perceived in extrafoveal vision.

The influence of social and symbolic cues on observers' gaze behaviour

Research has shown that social and symbolic cues presented in isolation and at fixation have strong effects on observers, but it is unclear how cues compare when they are presented away from fixation and embedded in natural scenes. We here compare the effects of two types of social cue (gaze and pointing gestures) and one type of symbolic cue (arrow signs) on eye movements of observers under two viewing conditions (free viewing vs. a memory task). The results suggest that social cues are looked at more quickly, for longer and more frequently than the symbolic arrow cues. An analysis of saccades initiated from the cue suggests that the pointing cue leads to stronger cueing than the gaze and the arrow cue. While the task had only a weak influence on gaze orienting to the cues, stronger cue following was found for free viewing compared to the memory task.

Eye gaze and head orientation modulate the inhibition of return for faces

The present study used an inhibition of return (IOR) spatial cueing paradigm to examine how gaze direction and head orientation modulate attention capture for human faces. Target response time (RT) was measured after the presentation of a peripheral cue, which was either a face (with front-facing or averted gaze, in either frontal head view or averted head view) or a house (control). Participants fixated on a centered cross at all times and responded via button press to a peripheral target after a variable stimulus onset asynchrony (SOA) from the stimulus cue. At the shortest SOA (150 ms), RTs were shorter for faces than houses, independent of an IOR response, suggesting a cue-based RT advantage elicited by faces. At the longest SOA (2,400 ms), a larger IOR magnitude was found for faces compared to houses. Both the cue-based RT advantage and later IOR responses were modulated by gaze-head congruency; these effects were strongest for frontal gaze faces in frontal head view, and for averted gaze faces in averted head view. Importantly, participants were not given any specific information regarding the stimuli, nor were they told the true purpose of the study. These findings indicate that the congruent combination of head and gaze direction influence the exogenous attention capture of faces during inhibition of return.

Viewing Complex, Dynamic Scenes "Through the Eyes" of Another Person: The Gaze-Replay Paradigm

We present a novel “Gaze-Replay” paradigm that allows the experimenter to directly test how particular patterns of visual input—generated from people’s actual gaze patterns—influence the interpretation of the visual scene. Although this paradigm can potentially be applied across domains, here we applied it specifically to social comprehension. Participants viewed complex, dynamic scenes through a small window displaying only the foveal gaze pattern of a gaze “donor.” This was intended to simulate the donor’s visual selection, such that a participant could effectively view scenes “through the eyes” of another person. Throughout the presentation of scenes presented in this manner, participants completed a social comprehension task, assessing their abilities to recognize complex emotions. The primary aim of the study was to assess the viability of this novel approach by examining whether these Gaze-Replay windowed stimuli contain sufficient and meaningful social information for the viewer to complete this social perceptual and cognitive task. The results of the study suggested this to be the case; participants performed better in the Gaze-Replay condition compared to a temporally disrupted control condition, and compared to when they were provided with no visual input. This approach has great future potential for the exploration of experimental questions aiming to unpack the relationship between visual selection, perception, and cognition.

Social decisions affect neural activity to perceived dynamic gaze

Gaze direction, a cue of both social and spatial attention, is known to modulate early neural responses to faces e.g. N170. However, findings in the literature have been inconsistent, likely reflecting differences in stimulus characteristics and task requirements. Here, we investigated the effect of task on neural responses to dynamic gaze changes: away and toward transitions (resulting or not in eye contact). Subjects performed, in random order, social (away/toward them) and non-social (left/right) judgment tasks on these stimuli. Overall, in the non-social task, results showed a larger N170 to gaze aversion than gaze motion toward the observer. In the social task, however, this difference was no longer present in the right hemisphere, likely reflecting an enhanced N170 to gaze motion toward the observer. Our behavioral and event-related potential data indicate that performing social judgments enhances saliency of gaze motion toward the observer, even those that did not result in gaze contact. These data and that of previous studies suggest two modes of processing visual information: a 'default mode' that may focus on spatial information; a 'socially aware mode' that might be activated when subjects are required to make social judgments. The exact mechanism that allows switching from one mode to the other remains to be clarified.

The Mona Lisa effect: neural correlates of centered and off-centered gaze

The Mona Lisa effect describes the phenomenon when the eyes of a portrait appear to look at the observer regardless of the observer's position. Recently, the metaphor of a cone of gaze has been proposed to describe the range of gaze directions within which a person feels looked at. The width of the gaze cone is about five degrees of visual angle to either side of a given gaze direction. We used functional magnetic resonance imaging to investigate how the brain regions involved in gaze direction discrimination would differ between centered and decentered presentation positions of a portrait exhibiting eye contact. Subjects observed a given portrait's eyes. By presenting portraits with varying gaze directions-eye contact (0°), gaze at the edge of the gaze cone (5°), and clearly averted gaze (10°), we revealed that brain response to gaze at the edge of the gaze cone was similar to that produced by eye contact and different from that produced by averted gaze. Right fusiform gyrus and right superior temporal sulcus showed stronger activation when the gaze was averted as compared to eye contact. Gaze sensitive areas, however, were not affected by the portrait's presentation location. In sum, although the brain clearly distinguishes averted from centered gaze, a substantial change of vantage point does not alter neural activity, thus providing a possible explanation why the feeling of eye contact is upheld even in decentered stimulus positions.

Deceptive body movements reverse spatial cueing in soccer

The purpose of the experiments was to analyse the spatial cueing effects of the movements of soccer players executing normal and deceptive (step-over) turns with the ball. Stimuli comprised normal resolution or point-light video clips of soccer players dribbling a football towards the observer then turning right or left with the ball. Clips were curtailed before or on the turn (-160, -80, 0 or +80 ms) to examine the time course of direction prediction and spatial cueing effects. Participants were divided into higher-skilled (HS) and lower-skilled (LS) groups according to soccer experience. In experiment 1, accuracy on full video clips was higher than on point-light but results followed the same overall pattern. Both HS and LS groups correctly identified direction on normal moves at all occlusion levels. For deceptive moves, LS participants were significantly worse than chance and HS participants were somewhat more accurate but nevertheless substantially impaired. In experiment 2, point-light clips were used to cue a lateral target. HS and LS groups showed faster reaction times to targets that were congruent with the direction of normal turns, and to targets incongruent with the direction of deceptive turns. The reversed cueing by deceptive moves coincided with earlier kinematic events than cueing by normal moves. It is concluded that the body kinematics of soccer players generate spatial cueing effects when viewed from an opponent's perspective. This could create a reaction time advantage when anticipating the direction of a normal move. A deceptive move is designed to turn this cueing advantage into a disadvantage. Acting on the basis of advance information, the presence of deceptive moves primes responses in the wrong direction, which may be only partly mitigated by delaying a response until veridical cues emerge.

A cognitive ethology study of first- and third-person perspectives

The aim of the present study was to test the cognitive ethology approach, which seeks to link cognitions and behaviours as they operate in everyday life with those studied in controlled lab-based investigations. Our test bed was the understanding of first-person and third-person perspectives, which in lab-based investigations have been defined in a diverse and multi-faceted manner. We hypothesized that because these lab-based investigations seek to connect with how first- and third-person perspective operates in everyday life, then either some of the divergent lab-based definitions are missing their mark or the everyday conceptualization of first- and third-person perspective is multi-faceted. Our investigation revealed the latter. By applying a cognitive ethology approach we were able to determine that a) peoples’ everyday understanding of perspective is diverse yet reliable, and b) a lab-based investigation that applies these diverse understandings in a controlled setting can accurately predict how people will perform. These findings provide a ‘proof of concept’ for the cognitive ethology approach. Moreover, the present data demonstrate that previous lab-based studies, that often had very different understandings of first- and third-person perspective, were each in and of themselves valid. That is, each is capturing part of a broader understanding of perspective in everyday life. Our results also revealed a novel social factor not included in traditional conceptualizations of first-person third-perspective, that of eye gaze, i.e., eye contact is equated strongly with first-person perspective and the lack of eye-contact with third-person perspective.

Perception of direct gaze does not require focus of attention

Previous research using averted (e.g., leftward or rightward) gaze indicates that gaze perception requires a focus of attention. However, direct gaze, compared with averted gaze, is processed in the brain preferentially and enhances cognitive functions. Thus, it is necessary to use direct gaze to investigate whether gaze perception is possible without focused attention. We conducted a dual-task paradigm in which attention was drawn away from gaze. Results showed performance on gaze-direction discrimination (direct vs. averted gaze) in the dual-task condition was only slightly lower than in the single-task condition; participants were able to discriminate direct from averted gaze without focusing their attention in a similar manner to when they did focus their attention. In contrast, when participants discriminated between averted gazes (leftward and rightward), performance dropped to near-chance levels. It was concluded that gaze perception does not require a focus of attention for direct gaze.

Effects of Peripheral Eccentricity and Head Orientation on Gaze Discrimination

Visual search tasks support a special role for direct gaze in human cognition, while classic gaze judgment tasks suggest the congruency between head orientation and gaze direction plays a central role in gaze perception. Moreover, whether gaze direction can be accurately discriminated in the periphery using covert attention is unknown. In the present study, individual faces in frontal and in deviated head orientations with a direct or an averted gaze were flashed for 150 ms across the visual field; participants focused on a centred fixation while judging the gaze direction. Gaze discrimination speed and accuracy varied with head orientation and eccentricity. The limit of accurate gaze discrimination was less than ±6° eccentricity. Response times suggested a processing facilitation for direct gaze in fovea, irrespective of head orientation, however, by ±3° eccentricity, head orientation started biasing gaze judgments, and this bias increased with eccentricity. Results also suggested a special processing of frontal heads with direct gaze in central vision, rather than a general congruency effect between eye and head cues. Thus, while both head and eye cues contribute to gaze discrimination, their role differs with eccentricity.

What interests them in the pictures?--differences in eye-tracking between rhesus monkeys and humans

Studies estimating eye movements have demonstrated that non-human primates have fixation patterns similar to humans at the first sight of a picture. In the current study, three sets of pictures containing monkeys, humans or both were presented to rhesus monkeys and humans. The eye movements on these pictures by the two species were recorded using a Tobii eye-tracking system. We found that monkeys paid more attention to the head and body in pictures containing monkeys, whereas both monkeys and humans paid more attention to the head in pictures containing humans. The humans always concentrated on the eyes and head in all the pictures, indicating the social role of facial cues in society. Although humans paid more attention to the hands than monkeys, both monkeys and humans were interested in the hands and what was being done with them in the pictures. This may suggest the importance and necessity of hands for survival. Finally, monkeys scored lower in eye-tracking when fixating on the pictures, as if they were less interested in looking at the screen than humans. The locations of fixation in monkeys may provide insight into the role of eye movements in an evolutionary context.

Information and repetition change children's visual strategies when viewing magic tricks with and without gaze cues

Gaze cues and direct gaze attract visual attention. However, few studies have explored visual cues in children within realistic contexts. The effect of information and repetitive stimulus presentation has not been thoroughly studied with dynamic stimuli. The aim of the present study was to investigate how information affects the visual strategies of children measured by the number of fixations on certain areas of interest and their durations. Furthermore, this study examined the effect of gaze cues and direct gaze. In two consecutive experiments, children's visual strategies when viewing magic tricks were measured by an eye tracker. Gaze cues were only present in Experiment 1.The results showed that repetitive stimulus presentation and information caused children to change their visual strategies when viewing magic tricks with and without gaze cues. However, the effect was larger when the gaze cues were not present. These findings in children were similar to those in adults.

Category Processing and the human likeness dimension of the Uncanny Valley Hypothesis: Eye-Tracking Data

The Uncanny Valley Hypothesis (Mori, 1970) predicts that perceptual difficulty distinguishing between a humanlike object (e.g., lifelike prosthetic hand, mannequin) and its human counterpart evokes negative affect. Research has focused on affect, with inconsistent results, but little is known about how objects along the hypothesis' dimension of human likeness (DHL) are actually perceived. This study used morph continua based on human and highly realistic computer-generated (avatar) faces to represent the DHL. Total number and dwell time of fixations to facial features were recorded while participants (N = 60) judged avatar versus human category membership of the faces in a forced choice categorization task. Fixation and dwell data confirmed the face feature hierarchy (eyes, nose, and mouth in this order of importance) across the DHL. There were no further findings for fixation. A change in the relative importance of these features was found for dwell time, with greater preferential processing of eyes and mouth of categorically ambiguous faces compared with unambiguous avatar faces. There were no significant differences between ambiguous and human faces. These findings applied for men and women, though women generally dwelled more on the eyes to the disadvantage of the nose. The mouth was unaffected by gender. In summary, the relative importance of facial features changed on the DHL's non-human side as a function of categorization ambiguity. This change was indicated by dwell time only, suggesting greater depth of perceptual processing of the eyes and mouth of ambiguous faces compared with these features in unambiguous avatar faces.

Importance of the inverted control in measuring holistic face processing with the composite effect and part-whole effect

Holistic coding for faces is shown in several illusions that demonstrate integration of the percept across the entire face. The illusions occur upright but, crucially, not inverted. Converting the illusions into experimental tasks that measure their strength – and thus index degree of holistic coding – is often considered straightforward yet in fact relies on a hidden assumption, namely that there is no contribution to the experimental measure from secondary cognitive factors. For the composite effect, a relevant secondary factor is size of the “spotlight” of visuospatial attention. The composite task assumes this spotlight can be easily restricted to the target half (e.g., top-half) of the compound face stimulus. Yet, if this assumption were not true then a large spotlight, in the absence of holistic perception, could produce a false composite effect, present even for inverted faces and contributing partially to the score for upright faces. We review evidence that various factors can influence spotlight size: race/culture (Asians often prefer a more global distribution of attention than Caucasians); sex (females can be more global); appearance of the join or gap between face halves; and location of the eyes, which typically attract attention. Results from five experiments then show inverted faces can sometimes produce large false composite effects, and imply that whether this happens or not depends on complex interactions between causal factors. We also report, for both identity and expression, that only top-half face targets (containing eyes) produce valid composite measures. A sixth experiment demonstrates an example of a false inverted part-whole effect, where encoding-specificity is the secondary cognitive factor. We conclude the inverted face control should be tested in all composite and part-whole studies, and an effect for upright faces should be interpreted as a pure measure of holistic processing only when the experimental design produces no effect inverted.

Attentional shifts by gaze direction in voluntary orienting: evidence from a microsaccade study

Shifts in spatial attention can be induced by the gaze direction of another. However, it is unclear whether gaze direction influences the allocation of attention by reflexive or voluntary orienting. The present study was designed to examine which type of attentional orienting is elicited by gaze direction. We conducted two experiments to answer this question. In Experiment 1, we used a modified Posner paradigm with gaze cues and measured microsaccades to index the allocation of attention. We found that microsaccade direction followed cue direction between 200 and 400 ms after gaze cues were presented. This is consistent with the latencies observed in other microsaccade studies in which voluntary orienting is manipulated, suggesting that gaze direction elicits voluntary orienting. However, Experiment 1 did not separate voluntary and reflexive orienting directionally, so in Experiment 2, we used an anticue task in which cue direction (direction to allocate attention) was the opposite of gaze direction (direction of gaze in depicted face). The results in Experiment 2 were consistent with those from Experiment 1. Microsaccade direction followed the cue direction, not gaze direction. Taken together, these results indicate that the shift in spatial attention elicited by gaze direction is voluntary orienting.

The anti-orienting phenomenon revisited: effects of gaze cues on antisaccade performance

When the eye gaze of a face is congruent with the direction of an upcoming target, saccadic eye movements of the observer towards that target are generated more quickly, in comparison with eye gaze incongruent with the direction of the target. This work examined the conflict in an antisaccade task, when eye gaze points towards the target, but the saccadic eye movement should be triggered in the opposite direction. In a gaze cueing paradigm, a central face provided an attentional gaze cue towards the target or away from the target. Participants (N = 38) generated pro- and antisaccades to peripheral targets that were congruent or incongruent with the previous gaze cue. Paradoxically, facilitatory effects of a gaze cue towards the target were observed for both the pro- and antisaccade tasks. The results are consistent with the idea that eye gaze cues are processed in the task set that is compatible with the saccade programme. Thus, in an antisaccade paradigm, participants may anti-orient with respect to the gaze cue, resulting in faster saccades on trials when the gaze cue is towards the target. The results resemble a previous observation by Fischer and Weber (Exp Brain Res 109:507-512, 1996) using low-level peripheral cues. The current study extends this finding to include central socially communicative cues.

Spatial orienting of attention simultaneously cued by automatic social and nonsocial cues

The appearance of a stimulus in the periphery and the direction of another person's eye gaze have both been shown to automatically orient attention toward the stimulus and the gazed-at location, respectively. In the present experiment, we examined the effects of viewing both a peripheral stimulus and an eye gaze stimulus simultaneously in order to determine whether one is "more automatic" (i.e., faster, dominates) than the other and whether the two processes interact. Using a spatial cueing paradigm, we measured latency of localization of a target stimulus that was validly or invalidly cued by an uninformative (i.e., nonpredictive) peripheral cue, an uninformative eye gaze cue, or both simultaneously (double cue). We included a short and a long cue-target interval in order to investigate the early and late facilitatory and inhibitory effects of the two processes. Results demonstrated that when the double cues were consistent with each other (indicating the same target location), the effects, both early and late, were the same as when the peripheral cue was presented alone. When the double cues were inconsistent (indicating opposite target locations), the late effect was the same as the peripheral cue, but the early effect was intermediate between the two types of cues. Our results better support an interactive, rather than an additive relationship between social and nonsocial automatic orienting. The double cue conditions that showed similar effects to the peripheral cues suggest that the peripheral cue dominates.

Visual cognition during real social interaction

Laboratory studies of social visual cognition often simulate the critical aspects of joint attention by having participants interact with a computer-generated avatar. Recently, there has been a movement toward examining these processes during authentic social interaction. In this review, we will focus on attention to faces, attentional misdirection, and a phenomenon we have termed social inhibition of return (Social IOR), that have revealed aspects of social cognition that were hitherto unknown. We attribute these discoveries to the use of paradigms that allow for more realistic social interactions to take place. We also point to an area that has begun to attract a considerable amount of interest-that of Theory of Mind (ToM) and automatic perspective taking-and suggest that this too might benefit from adopting a similar approach.

Dog experts' brains distinguish socially relevant body postures similarly in dogs and humans

We read conspecifics' social cues effortlessly, but little is known about our abilities to understand social gestures of other species. To investigate the neural underpinnings of such skills, we used functional magnetic resonance imaging to study the brain activity of experts and non-experts of dog behavior while they observed humans or dogs either interacting with, or facing away from a conspecific. The posterior superior temporal sulcus (pSTS) of both subject groups dissociated humans facing toward each other from humans facing away, and in dog experts, a distinction also occurred for dogs facing toward vs. away in a bilateral area extending from the pSTS to the inferior temporo-occipital cortex: the dissociation of dog behavior was significantly stronger in expert than control group. Furthermore, the control group had stronger pSTS responses to humans than dogs facing toward a conspecific, whereas in dog experts, the responses were of similar magnitude. These findings suggest that dog experts' brains distinguish socially relevant body postures similarly in dogs and humans.

Social attention with real versus reel stimuli: toward an empirical approach to concerns about ecological validity

Cognitive neuroscientists often study social cognition by using simple but socially relevant stimuli, such as schematic faces or images of other people. Whilst this research is valuable, important aspects of genuine social encounters are absent from these studies, a fact that has recently drawn criticism. In the present review we argue for an empirical approach to the determination of the equivalence of different social stimuli. This approach involves the systematic comparison of different types of social stimuli ranging in their approximation to a real social interaction. In garnering support for this cognitive ethological approach, we focus on recent research in social attention that has involved stimuli ranging from simple schematic faces to real social interactions. We highlight both meaningful similarities and differences in various social attentional phenomena across these different types of social stimuli thus validating the utility of the research initiative. Furthermore, we argue that exploring these similarities and differences will provide new insights into social cognition and social neuroscience.