Dual-route model of the effect of head orientation on perceived gaze direction

The Timing of Gaze Direction Perception: ERP Decoding and Task Modulation

Distinguishing the direction of another person's eye gaze is extremely important in everyday social interaction, as it provides critical information about people's attention and, therefore, intentions. The temporal dynamics of gaze processing have been investigated using event-related potentials (ERPs) recorded with electroencephalography (EEG). However, the moment at which our brain distinguishes the gaze direction (GD), irrespectively of other facial cues, remains unclear. To solve this question, the present study aimed to investigate the time course of gaze direction processing, using an ERP decoding approach, based on the combination of a support vector machine and error-correcting output codes. We recorded EEG in young healthy subjects, 32 of them performing GD detection and 34 conducting face orientation tasks. Both tasks presented 3D realistic faces with five different head and gaze orientations each: 30°, 15° to the left or right, and 0°. While the classical ERP analyses did not show clear GD effects, ERP decoding analyses revealed that discrimination of GD, irrespective of head orientation, started at 140 ms in the GD task and at 120 ms in the face orientation task. GD decoding accuracy was higher in the GD task than in the face orientation task and was the highest for the direct gaze in both tasks. These findings suggest that the decoding of brain patterns is modified by task relevance, which changes the latency and the accuracy of GD decoding.

The integration of head and body cues during the perception of social interactions

Humans spend a large proportion of time participating in social interactions. The ability to accurately detect and respond to human interactions is vital for social functioning, from early childhood through to older adulthood. This detection ability arguably relies on integrating sensory information from the interactants. Within the visual modality, directional information from a person's eyes, head, and body are integrated to inform where another person is looking and who they are interacting with. To date, social cue integration research has focused largely on the perception of isolated individuals. Across two experiments, we investigated whether observers integrate body information with head information when determining whether two people are interacting, and manipulated frame of reference (one of the interactants facing observer vs. facing away from observer) and the eye-region visibility of the interactant. Results demonstrate that individuals integrate information from the body with head information when perceiving dyadic interactions, and that integration is influenced by the frame of reference and visibility of the eye-region. Interestingly, self-reported autistics traits were associated with a stronger influence of body information on interaction perception, but only when the eye-region was visible. This study investigated the recognition of dyadic interactions using whole-body stimuli while manipulating eye visibility and frame of reference, and provides crucial insights into social cue integration, as well as how autistic traits affect cue integration, during perception of social interactions.

Autistic traits modulate the influence of face masks on gaze perception

Detecting when others are looking at us is a crucial social skill. Accordingly, a range of gaze angles is perceived as self-directed; this is termed the "cone of direct gaze" (CoDG). Multiple cues, such as nose and head orientation, are integrated during gaze perception. Thus, occluding the lower portion of the face, such as with face masks during the COVID-19 pandemic, may influence how gaze is perceived. Individual differences in the prioritisation of eye-region and non-eye-region cues may modulate the influence of face masks on gaze perception. Autistic individuals, who may be more reliant on non-eye-region directional cues during gaze perception, might be differentially affected by face masks. In the present study, we compared the CoDG when viewing masked and unmasked faces (N = 157) and measured self-reported autistic traits. The CoDG was wider for masked compared to unmasked faces, suggesting that reduced reliability of lower face cues increases the range of gaze angles perceived as self-directed. Additionally, autistic traits positively predicted the magnitude of CoDG difference between masked and unmasked faces. This study provides crucial insights into the effect of face masks on gaze perception, and how they may affect autistic individuals to a greater extent.

Inferential eye movement control while following dynamic gaze

Attending to other people's gaze is evolutionary important to make inferences about intentions and actions. Gaze influences covert attention and triggers eye movements. However, we know little about how the brain controls the fine-grain dynamics of eye movements during gaze following. Observers followed people's gaze shifts in videos during search and we related the observer eye movement dynamics to the time course of gazer head movements extracted by a deep neural network. We show that the observers' brains use information in the visual periphery to execute predictive saccades that anticipate the information in the gazer's head direction by 190-350ms. The brain simultaneously monitors moment-to-moment changes in the gazer's head velocity to dynamically alter eye movements and re-fixate the gazer (reverse saccades) when the head accelerates before the initiation of the first forward gaze-following saccade. Using saccade-contingent manipulations of the videos, we experimentally show that the reverse saccades are planned concurrently with the first forward gaze-following saccade and have a functional role in reducing subsequent errors fixating on the gaze goal. Together, our findings characterize the inferential and functional nature of social attention's fine-grain eye movement dynamics.

Head and body cues guide eye movements and facilitate target search in real-world videos

Static gaze cues presented in central vision result in observer shifts of covert attention and eye movements, and benefits in perceptual performance in the detection of simple targets. Less is known about how dynamic gazer behaviors with head and body motion influence search eye movements and performance in perceptual tasks in real-world scenes. Participants searched for a target person (yes/no task, 50% presence), whereas watching videos of one to three gazers looking at a designated person (50% valid gaze cue, looking at the target). To assess the contributions of different body parts, we digitally erase parts of the gazers in the videos to create three different body parts/whole conditions for gazers: floating heads (only head movements), headless bodies (only lower body movements), and the baseline condition with intact head and body. We show that valid dynamic gaze cues guided participants' eye movements (up to 3 fixations) closer to the target, speeded the time to foveate the target, reduced fixations to the gazers, and improved target detection. The effect of gaze cues in guiding eye movements to the search target was the smallest when the gazer's head was removed from the videos. To assess the inherent information about gaze goal location for each body parts/whole condition, we collected perceptual judgments estimating gaze goals by a separate group of observers with unlimited time. Observers' perceptual judgments showed larger estimate errors when the gazer's head was removed. This suggests that the reduced eye movement guidance from lower body cueing is related to observers' difficulty extracting gaze information without the presence of the head. Together, the study extends previous work by evaluating the impact of dynamic gazer behaviors on search with videos of real-world cluttered scenes.

Gaze perception from head and pupil rotations in 2D and 3D: Typical development and the impact of autism spectrum disorder

The study of gaze perception has largely focused on a single cue (the eyes) in two-dimensional settings. While this literature suggests that 2D gaze perception is shaped by atypical development, as in Autism Spectrum Disorder (ASD), gaze perception is in reality contextually-sensitive, perceived as an emergent feature conveyed by the rotation of the pupils and head. We examined gaze perception in this integrative context, across development, among children and adolescents developing typically or with ASD with both 2D and 3D stimuli. We found that both groups utilized head and pupil rotations to judge gaze on a 2D face. But when evaluating the gaze of a physically-present, 3D robot, the same ASD observers used eye cues less than their typically-developing peers. This demonstrates that emergent gaze perception is a slowly developing process that is surprisingly intact, albeit weakened in ASD, and illustrates how new technology can bridge visual and clinical science.

Spatial selectivity in adaptation to gaze direction

A person's focus of attention is conveyed by the direction of their eyes and face, providing a simple visual cue fundamental to social interaction. A growing body of research examines the visual mechanisms that encode the direction of another person's gaze as we observe them. Here we investigate the spatial receptive field properties of these mechanisms, by testing the spatial selectivity of sensory adaptation to gaze direction. Human observers were adapted to faces with averted gaze presented in one visual hemifield, then tested in their perception of gaze direction for faces presented in the same or opposite hemifield. Adaptation caused strong, repulsive perceptual aftereffects, but only for faces presented in the same hemifield as the adapter. This occurred even though adapting and test stimuli were in the same external location across saccades. Hence, there was clear evidence for retinotopic adaptation and a relative lack of either spatiotopic or spatially invariant adaptation. These results indicate that adaptable representations of gaze direction in the human visual system have retinotopic spatial receptive fields. This strategy of coding others' direction of gaze with positional specificity relative to one's own eye position may facilitate key functions of gaze perception, such as socially cued shifts in visual attention.

Holistic processing of gaze cues during interocular suppression

Direct eye contact is preferentially processed over averted gaze and has been shown to gain privileged access to conscious awareness during interocular suppression. This advantage might be driven by local features associated with direct gaze, such as the amount of visible sclera. Alternatively, a holistic representation of gaze direction, which depends on the integration of head and eye information, might drive the effects. Resolving this question is interesting because it speaks to whether the processing of higher-level social information in the visual system, such as facial characteristics that rely on holistic processing, is dependent on conscious awareness. The Wollaston Illusion is a visual illusion that allows researchers to manipulate perceived gaze direction while keeping local eye features constant. Here we used this illusion to elucidate the driving factor facilitating the direct gaze advantage during interocular suppression. Using continuous flash suppression, we rendered Wollaston faces with direct and averted gaze (initially) invisible. These faces conveyed different gaze directions but contained identical eye regions. Our results showed clear evidence for a direct gaze advantage with Wollaston faces, indicating that holistic representations of gaze direction may drive the direct gaze advantage during interocular suppression.

Recognizing Students and Detecting Student Engagement with Real-Time Image Processing

With COVID-19, formal education was interrupted in all countries and the importance of distance learning has increased. It is possible to teach any lesson with various communication tools but it is difficult to know how far this lesson reaches to the students. In this study, it is aimed to monitor the students in a classroom or in front of the computer with a camera in real time, recognizing their faces, their head poses, and scoring their distraction to detect student engagement based on their head poses and Eye Aspect Ratios. Distraction was determined by associating the students’ attention with looking at the teacher or the camera in the right direction. The success of the face recognition and head pose estimation was tested by using the UPNA Head Pose Database and, as a result of the conducted tests, the most successful result in face recognition was obtained with the Local Binary Patterns method with a 98.95% recognition rate. In the classification of student engagement as Engaged and Not Engaged, support vector machine gave results with 72.4% accuracy. The developed system will be used to recognize and monitor students in the classroom or in front of the computer, and to determine the course flow autonomously.

Gaze-cued shifts of attention and microsaccades are sustained for whole bodies but are transient for body parts

Gaze direction is an evolutionarily important mechanism in daily social interactions. It reflects a person’s internal cognitive state, spatial locus of interest, and predicts future actions. Studies have used static head images presented foveally and simple synthetic tasks to find that gaze orients attention and facilitates target detection at the cued location in a sustained manner. Little is known about how people’s natural gaze behavior, including eyes, head, and body movements, jointly orient covert attention, microsaccades, and facilitate performance in more ecological dynamic scenes. Participants completed a target person detection task with videos of real scenes. The videos showed people looking toward (valid cue) or away from a target (invalid cue) location. We digitally manipulated the individuals in the videos directing gaze to create three conditions: whole-intact (head and body movements), floating heads (only head movements), and headless bodies (only body movements). We assessed their impact on participants’ behavioral performance and microsaccades during the task. We show that, in isolation, an individual’s head or body orienting toward the target-person direction led to facilitation in detection that is transient in time (200 ms). In contrast, only the whole-intact condition led to sustained facilitation (500 ms). Furthermore, observers executed microsaccades more frequently towards the cued direction for valid trials, but this bias was sustained in time only with the joint presence of head and body parts. Together, the results differ from previous findings with foveally presented static heads. In more real-world scenarios and tasks, sustained attention requires the presence of the whole-intact body of the individuals dynamically directing their gaze.

Atypical processing pattern of gaze cues in dynamic situations in autism spectrum disorders

Psychological studies have generally shown that individuals with Autism Spectrum Disorder (ASD) have particularity in the processing of social information by using static or abstract images. Yet, a recent study showed that there was no difference in their use of social or non-social cues in dynamic interactive situations. To establish the cause of the inconsistent results, we added gaze cues in different directions to the chase detection paradigm to explore whether they would affect the performance of participants with ASD. Meanwhile, eye-tracking methodology was used to investigate whether the processing patterns of gaze cues were different between individuals with ASD and TD. In this study, unlike typical controls, participants with ASD showed no detection advantage when the direction of gaze was consistent with the direction of movement (oriented condition). The results suggested that individuals with ASD may utilize an atypical processing pattern, which makes it difficult for them to use social information contained in oriented gaze cues in dynamic interactive situations.

Is there a 'zone of eye contact' within the borders of the face?

Eye contact is a salient feature of everyday interactions, yet it is not obvious what the physical conditions are under which we feel that we have eye contact with another person. Here we measure the range of locations that gaze can fall on a person's face to elicit a sense of eye contact. Participants made judgements about eye contact while viewing rendered images of faces with finely-varying gaze direction at a close interpersonal distance (50 cm). The 'zone of eye contact' tends to peak between the two eyes and is often surprisingly narrower than the observer's actual eye region. Indeed, the zone tends to extend further across the face in height than in width. This shares an interesting parallel with the 'cyclopean eye' of visual perspective - our sense of looking out from a single point in space despite the physical separation of our two eyes. The distribution of eye-contact strength across the face can be modelled at the individual-subject level as a 2D Gaussian function. Perception of eye contact is more precise than the sense of having one's face looked at, which captures a wider range of gaze locations in both the horizontal and vertical dimensions, at least at the close viewing distance used in the present study. These features of eye-contact perception are very similar cross-culturally, tested here in Australian and Japanese university students. However, the shape and position of the zone of eye contact does vary depending on recent sensory experience: adaptation to faces with averted gaze causes a pronounced shift and widening of the zone across the face, and judgements about eye contact also show a positive serial dependence. Together, these results provide insight into the conditions under which eye contact is felt, with respect to face morphology, culture, and sensory context.

Inverting the Wollaston Illusion: Gaze Direction Attracts Perceived Head Orientation

In the early 19th century, William H. Wollaston impressed the Royal Society of London with engravings of portraits. He manipulated facial features, such as the nose, and thereby dramatically changed the perceived gaze direction, although the eye region with iris and eye socket had remained unaltered. This Wollaston illusion can be thought of as head orientation attracting perceived gaze direction when the eye region is unchanged. In naturalistic viewing, the eye region changes with head orientation and typically produces a repulsion effect. Here we explore if there is a flip side to the illusion. Does the gaze direction also alter the perceived direction of the head? We used copies of the original drawings and a computer-rendered avatar as stimuli. Gaze direction does indeed alter perceived head orientation. Perceived head orientation is biased toward the direction of gaze.

Examining Gaze Cone Shape and Size

Another person's looking behavior is used by observers to judge gaze direction and fixation points. An important task in this context is the judgement of direct gaze, that is, the perception of being looked at. The cone of gaze can be defined as the range of fixation points that support direct gaze. The cone concept implies that this range lawfully increases with distance, but that the cone angle is constant. The present experiment tested the concept with a larger number and a more extended range of distances than previously done, and took care of possible directional errors. The gaze cone was found to be roughly linear, and stable between 1.6 m and 7.9 m - an almost perfect cone. The mean cone size subtended 5.2° in diameter when averaged over ascending and descending series. Measures differed, however, in ascending and descending series, consistent with a conservative bias. Also, the variability of judgements increased slightly with distance. Results are discussed considering whether cone size is actually smaller than often reported in the literature.

Face Pareidolia Recruits Mechanisms for Detecting Human Social Attention

Face pareidolia is the phenomenon of seeing facelike structures in everyday objects. Here, we tested the hypothesis that face pareidolia, rather than being limited to a cognitive or mnemonic association, reflects the activation of visual mechanisms that typically process human faces. We focused on sensory cues to social attention, which engage cell populations in temporal cortex that are susceptible to habituation effects. Repeated exposure to “pareidolia faces” that appear to have a specific direction of attention causes a systematic bias in the perception of where human faces are looking, indicating that overlapping sensory mechanisms are recruited when we view human faces and when we experience face pareidolia. These cross-adaptation effects are significantly reduced when pareidolia is abolished by removing facelike features from the objects. These results indicate that face pareidolia is essentially a perceptual phenomenon, occurring when sensory input is processed by visual mechanisms that have evolved to extract specific social content from human faces.

Quantifying the Wollaston Illusion

In the early 19th century, William H. Wollaston impressed the Royal Society of London with engravings of portraits. He manipulated facial features, such as the nose, and thereby dramatically changed the perceived gaze direction, although the eye region with iris and eye socket had remained unaltered. This Wollaston illusion has been replicated numerous times but never with the original stimuli. We took the eyes (pupil and iris) from Wollaston's most prominent engraving and measured their perceived gaze direction in an analog fashion. We then systematically added facial features (eye socket, eyebrows, nose, skull, and hair). These features had the power to divert perceived gaze direction by up to 20°, which confirms Wollaston's phenomenal observation. The effect can be thought of as an attractor effect, that is, cues that indicate a slight change in head orientation have the power to divert perceived gaze direction.

Deconstructing eye contact perception: Measuring perceptual precision and self-referential tendency using an online psychophysical eye contact detection task

Eye contact perception—the ability to accurately and efficiently discriminate others’ gaze directions—is critical to understanding others and functioning in a complex social world. Previous research shows that it is affected in multiple neuropsychiatric disorders accompanied by social dysfunction, and understanding the cognitive processes giving rise to eye contact perception would help advance mechanistic investigations of psychopathology. This study aims to validate an online, psychophysical eye contact detection task through which two constituent cognitive components of eye contact perception (perceptual precision and self-referential tendency) can be derived. Data collected from a large online sample showed excellent test-retest reliability for self-referential tendency and moderate reliability for perceptual precision. Convergence validity was supported by correlations with social cognitive measures tapping into different aspects of understanding others. Hierarchical regression analyses revealed that perceptual precision and self-referential tendency explained unique variance in social cognition, suggesting that they measure unique aspects of related constructs. Overall, this study provided support for the reliability and validity of the eye contact perception metrics derived using the online Eye Contact Detection Task. The value of the task for future psychopathology research was discussed.

Turning Heads: The Effects of Face View and Eye Gaze Direction on the Perceived Attractiveness of Expressive Faces

We investigated whether the perceived attractiveness of expressive faces was influenced by head turn and eye gaze towards or away from the observer. In all experiments, happy faces were consistently rated as more attractive than angry faces. A head turn towards the observer, whereby a full-face view was shown, was associated with relatively higher attractiveness ratings when gaze direction was aligned with face view (Experiment 1). However, preference for full-face views of happy faces was not affected by gaze shifts towards or away from the observer (Experiment 2a). In Experiment 3, the relative duration of each face view (front-facing or averted at 15°) during a head turn away or towards the observer was manipulated. There was benefit on attractiveness ratings for happy faces shown for a longer duration from the front view, regardless of the direction of head turn. Our findings support previous studies indicating a preference for positive expressions on attractiveness judgements, which is further enhanced by the front views of faces, whether presented during a head turn or shown statically. In sum, our findings imply a complex interaction between cues of social attention, indicated by the view of the face shown, and reward on attractiveness judgements of unfamiliar faces.

Gazing into space: Systematic biases in determining another's fixation distance from gaze vergence in upright and inverted faces

The eyes of others play a crucial role in social interactions, providing information such as the focus of another's attention and their current thoughts and emotions. Although much research has focused on understanding how we perceive gaze direction, little has been done on gaze vergence, which can potentially yield information about the distance of another's fixation. Here, we presented participants with synthetic faces in a stereoscopically simulated 3-D environment to determine the absolute fixation distance at which they perceived a face to be gazing. The results showed an underestimation in fixation distance for downward-averted gaze and a limit in discrimination of gaze vergence beyond 35 cm. For inverted faces, fixation distance for gaze vergence in the lower visual field (corresponding to the avatar's upwards gaze) was underestimated, suggesting that our bias to underestimate others' fixation distance may rely on a viewer-centered, egocentric representation of interpersonal space.

Disrupted Eye Gaze Perception as a Biobehavioral Marker of Social Dysfunction: An RDoC Investigation

Social dysfunction is an intractable problem in a wide spectrum of psychiatric illnesses, undermining patients' capacities for employment, independent living, and maintaining meaningful relationships. Identifying common markers of social impairment across disorders and understanding their mechanisms are prerequisites to developing targeted neurobiological treatments that can be applied productively across diagnoses and illness stages to improve functional outcome. This project focuses on eye gaze perception, the ability to accurately and efficiently discriminate others' gaze direction, as a potential biomarker of social functioning that cuts across psychiatric diagnoses. This premise builds on both the monkey and human literatures showing gaze perception as a basic building block supporting higher-level social communication and social development, and reports of abnormal gaze perception in multiple psychiatric conditions accompanied by prominent social dysfunction (e.g., psychosis-spectrum disorders, autism-spectrum disorders, social phobia). A large sample (n = 225) of adolescent and young adult (age 14-30) psychiatric patients (regardless of diagnosis) with various degrees of impaired social functioning, and demographically-matched healthy controls (n = 75) will be recruited for this study. Participant's psychiatric phenotypes, cognition, social cognition, and community functioning will be dimensionally characterized. Eye gaze perception will be assessed using a psychophysical task, and two metrics (precision, self-referential bias) that respectively tap into gaze perception disturbances at the visual perceptual and interpretation levels, independent of general deficits, will be derived using hierarchical Bayesian modeling. A subset of the participants (150 psychiatric patients, 75 controls) will additionally undergo multimodal fMRI to determine the functional and structural brain network features of altered gaze perception. The specific aims of this project are three-fold: (1) Determine the generality of gaze perception disturbances in psychiatric patients with prominent social dysfunction; (2) Map behavioral indices of gaze perception disturbances to dimensions of psychiatric phenotypes and core functional domains; and (3) Identify the neural correlates of altered gaze perception in psychiatric patients with social dysfunction. Successfully completing these specific aims will identify the specific basic deficits, clinical profile, and underlying neural circuits associated with social dysfunction that can be used to guide targeted, personalized treatments, thus advancing NIMH's Strategic Objective 1 (describe neural circuits associated with mental illnesses and map the connectomes for mental illnesses) and Objective 3 (develop new treatments based on discoveries in neuroscience and behavioral science).

The Effects of Hemianopia on Perception of Mutual Gaze

SIGNIFICANCE

Individuals with left hemianopic field loss (HFL), especially with neglect history, may have greater difficulties than individuals with right HFL in judging the direction of another person's gaze.

PURPOSE

Individuals with HFL often show a spatial bias in laboratory-based perceptual tasks. We investigated whether such biases also manifest in a more real-world task, perception of mutual gaze direction, an important, nonverbal communication cue in social interactions.

METHODS

Participants adjusted the eye position of a life-size virtual head on a monitor at a 1-m distance until (1) the eyes appeared to be looking straight at them, or (2) the eyes were perceived to be no longer looking at them (to the right and left).

RESULTS

Participants with right HFL (n = 8) demonstrated a rightward error in line bisection but made gaze judgments within the range of normally sighted controls (n = 17). Participants with left HFL without neglect history (n = 6) made leftward errors in line bisection and had more variable gaze judgments; three had estimates of gaze direction outside the reference range. Four participants with left HFL and neglect history made estimates of gaze direction that were to the right of the reference range.

CONCLUSIONS

Our results suggest that individuals with left HFL, especially with neglect history, may have greater difficulties than individuals with right HFL in compensating for low-level spatial biases (as manifested in line bisection) when performing the more complex, higher-level task of judging gaze direction.

Perceiving gaze from head and eye rotations: An integrative challenge for children and adults

Gaze is an emergent visual feature. A person's gaze direction is perceived not just based on the rotation of their eyes, but also their head. At least among adults, this integrative process appears to be flexible such that one feature can be weighted more heavily than the other depending on the circumstances. Yet it is unclear how this weighting might vary across individuals or across development. When children engage emergent gaze, do they prioritize cues from the head and eyes similarly to adults? Is the perception of gaze among individuals with autism spectrum disorder (ASD) emergent, or is it reliant on a single feature? Sixty adults (M = 29.86 years-of-age), thirty-seven typically developing children and adolescents (M = 9.3 years-of-age; range = 7-15), and eighteen children with ASD (M = 9.72 years-of-age; range = 7-15) viewed faces with leftward, rightward, or direct head rotations in conjunction with leftward or rightward pupil rotations, and then indicated whether the face was looking leftward or rightward. All individuals, across development and ASD status, used head rotation to infer gaze direction, albeit with some individual differences. However, the use of pupil rotation was heavily dependent on age. Finally, children with ASD used pupil rotation significantly less than typically developing (TD) children when inferring gaze direction, even after accounting for age. Our approach provides a novel framework for understanding individual and group differences in gaze as it is actually perceived-as an emergent feature. Furthermore, this study begins to address an important gap in ASD literature, taking the first look at emergent gaze perception in this population.

The Role of Emotional Expression and Eccentricity on Gaze Perception

The perception of another’s gaze direction and facial expression complements verbal communication and modulates how we interact with other people. However, our perception of these two cues is not always accurate, even when we are looking directly at the person. In addition, in many cases social communication occurs within groups of people where we can’t always look directly at every person in the group. Here, we sought to examine how the presence of other people influences our perception of a target face. We asked participants to judge the direction of gaze of the target face as either looking to their left, to their right or directly at them, when the face was viewed on its own or viewed within a group of other identity faces. The target face either had an angry or a neutral expression and was viewed directly (foveal experiment), or within peripheral vision (peripheral experiment). When the target was viewed within a group, the flanking faces also had either neutral or angry expressions and their gaze was in one of five different directions (from averted leftwards to averted rightwards in steps of 10°). When the target face was viewed foveally there was no effect of target emotion on participants’ judgments of its gaze direction. There was also no effect of the presence of flankers (regardless of expression) on the perception of the target gaze. When the target face was viewed peripherally, participants judged its direction of gaze to be direct over a wider range of gaze deviations than when viewed foveally, and more so for angry faces than neutral faces. We also find that flankers (regardless of emotional expression) did not influence performance. This suggests that observers judge that angry faces were looking at them over a broad range of gaze deviations in the periphery only, possibly resulting from increased uncertainty about the stimulus.

Assumptions about the positioning of virtual stimuli affect gaze direction estimates during Augmented Reality based interactions

We investigated gaze direction determination in dyadic interactions mediated by an Augmented Reality (AR) head-mounted-display. With AR, virtual content is overlaid on top of the real-world scene, offering unique data visualization and interaction opportunities. A drawback of AR however is related to uncertainty regarding the AR user’s focus of attention in social-collaborative settings: an AR user looking in our direction might either be paying attention to us or to augmentations positioned somewhere in between. In two psychophysical experiments, we assessed what impact assumptions concerning the positioning of virtual content attended by an AR user have on other people’s sensitivity to their gaze direction. In the first experiment we found that gaze discrimination was better when the participant was aware that the AR user was focusing on stimuli positioned on their depth plane as opposed to being positioned halfway between the AR user and the participant. In the second experiment, we found that this modulatory effect was explained by participants’ assumptions concerning which plane the AR user was focusing on, irrespective of these being correct. We discuss the significance of AR reduced gaze determination in social-collaborative settings as well as theoretical implications regarding the impact of this technology on social behaviour.

Task Dependent Effects of Head Orientation on Perceived Gaze Direction

The perception of gaze direction involves the integration of a number of sensory cues exterior to the eye-region. The orientation of the head is one such cue, which has an overall repulsive effect on the perceived direction of gaze. However, in a recent experiment, we found the measured effect of head orientation on perceived gaze direction differed within subjects, depending on whether a single- or two-interval task design was employed. This suggests a potential difference in the way the orientation of the head is integrated into the perception of gaze direction across tasks. Four experiments were conducted to investigate this difference. The first two experiments showed that the difference was not the result of some interaction between stimuli in the two-interval task, but rather, a difference between the types of judgment being made across tasks, where observers were making a directional (left/right) judgment in the single-interval task, and a non-directional (direct/indirect gaze) judgment in the two-interval task. A third experiment showed that this difference does not arise from observers utilizing a non-directional cue to direct gaze (the circularity of the pupil/iris) in making their non-directional judgments. The fourth experiment showed no substantial differences in the duration of evidence accumulation and processing between judgments, suggesting that observers are not integrating different sensory information across tasks. Together these experiments show that the sensory information from head orientation is flexibly weighted in the perception of gaze direction, and that the purpose of the observer, in sampling gaze information, can influence the consequent perception of gaze direction.

Perceptual integration of head and eye cues to gaze direction in schizophrenia

The perceptual mechanisms that underlie social experience in schizophrenia are increasingly becoming a target of empirical research. In the context of low-level vision, there is evidence for a reduction in the integration of sensory features in schizophrenia (e.g. increased thresholds for contour detection and motion coherence). In the context of higher-level vision, comparable differences in the integration of sensory features of the face could in theory impair the recognition of important social cues. Here we examine how the sense of where other people are looking relies upon the integration of eye-region cues and head-region cues. Adults with schizophrenia viewed face images designed to elicit the 'Wollaston illusion', a perceptual phenomenon in which the perceived gaze direction associated with a given pair of eyes is modulated by the surrounding sensory context. We performed computational modelling of these psychophysical data to quantify individual differences in the use of facial cues to gaze direction. We find that adults with schizophrenia exhibit a robust perceptual effect whereby their sense of other people's direction of gaze is strongly biased by sensory cues relating to head orientation in addition to eye region information. These results indicate that the visual integration of facial cues to gaze direction in schizophrenia is intact, helping to constrain theories of reduced integrative processing in higher-level and lower-level vision. In addition, robust gaze processing was evident in the tested participants despite reduced performance on a theory of mind task designed to assess higher-level social cognition.

Context Modulates Congruency Effects in Selective Attention to Social Cues

Head and gaze directions are used during social interactions as essential cues to infer where someone attends. When head and gaze are oriented toward opposite directions, we need to extract socially meaningful information despite stimulus conflict. Recently, a cognitive and neural mechanism for filtering-out conflicting stimuli has been identified while performing non-social attention tasks. This mechanism is engaged proactively when conflict is anticipated in a high proportion of trials and reactively when conflict occurs infrequently. Here, we investigated whether a similar mechanism is at play for limiting distraction from conflicting social cues during gaze or head direction discrimination tasks in contexts with different probabilities of conflict. Results showed that, for the gaze direction task only (Experiment 1), inverse efficiency (IE) scores for distractor-absent trials (i.e., faces with averted gaze and centrally oriented head) were larger (indicating worse performance) when these trials were intermixed with congruent/incongruent distractor-present trials (i.e., faces with averted gaze and tilted head in the same/opposite direction) relative to when the same distractor-absent trials were shown in isolation. Moreover, on distractor-present trials, IE scores for congruent (vs. incongruent) head-gaze pairs in blocks with rare conflict were larger than in blocks with frequent conflict, suggesting that adaptation to conflict was more efficient than adaptation to infrequent events. However, when the task required discrimination of head orientation while ignoring gaze direction, performance was not impacted by both block-level and current trial congruency (Experiment 2), unless the cognitive load of the task was increased by adding a concurrent task (Experiment 3). Overall, our study demonstrates that during attention to social cues proactive cognitive control mechanisms are modulated by the expectation of conflicting stimulus information at both the block- and trial-sequence level, and by the type of task and cognitive load. This helps to clarify the inherent differences in the distracting potential of head and gaze cues during speeded social attention tasks.

Perceived Object Trajectory Is Influenced by Others' Tracking Movements

In social interactions, it is highly salient to us where other people are looking. The ability to recover this information is critical to typical social development, helping us to coordinate our attention and behavior with others and understand their intentions and mental states [1-3]. The depth and direction in which another individual is fixating are specified jointly by their head position, eye deviation, and binocular vergence [4, 5]. It is hereto unknown, however, whether this dynamic visual information about others' focus of attention affects how we ourselves see the world. Here we show that the perceived depth and movement of physical objects in our environment are influenced by others' tracking behavior. This effect occurred even in the presence of conflicting size cues to object location and generalized to the context of apparent motion displays [6] and judgments about causal interactions between moving objects [7]. Perceived object trajectory was modulated primarily by the object-level motion of the tracking agent (e.g., the head), with less-pronounced effects of eye motion and low-level motion. Interestingly, comparable perceptual effects were induced by non-face objects that displayed similar tracking behavior, indicating a mechanism of distal coupling between the motion of the target and an appropriately moving inducer. These results demonstrate that social information can have a fundamental effect on our vision, such that the visual reality constructed in each brain is determined in part by what others see.

A bias-minimising measure of the influence of head orientation on perceived gaze direction

The orientation of the head is an important cue for gaze direction, and its role has been explained in a dual route model. The model incorporates both an attractive and a repulsive effect of head orientation, which act to support accurate gaze perception across large changes in natural stimuli. However, in all previous studies of which we are aware, measurements of the influence of head orientation on perceived gaze direction were obtained using a single-interval methodology, which may have been affected by response bias. Here we compare the single-interval methodology with a two-interval (bias-minimising) design. We find that although measures obtained using the two-interval design showed a stronger attractive effect of head orientation than previous studies, the influence of head orientation on perceived gaze direction still represents a genuine perceptual effect. Measurements obtained using the two-interval design were also shown to be more stable across sessions one week apart. These findings suggest the two-interval design should be used in future experiments, especially if comparing groups who may systematically differ in their biases, such as patients with schizophrenia or autism.

The center of attention: Metamers, sensitivity, and bias in the emergent perception of gaze

A person's gaze reveals much about their focus of attention and intentions. Sensitive perception of gaze is thus highly relevant for social interaction, especially when it is directed toward the viewer. Yet observers also tend to overestimate the likelihood that gaze is directed toward them. How might the visual system balance these competing goals, maximizing sensitivity for discriminating gazes that are relatively direct, while at the same time allowing many gazes to appear as if they look toward the viewer? Perceiving gaze is an emergent visual process that involves integrating information from the eyes with the rotation of the head. Here, we examined whether the visual system leverages emergent representation to balance these competing goals. We measured perceived gaze for a large range of pupil and head combinations and found that head rotation has a nonlinear influence on a person's apparent direction of looking, especially when pupil rotations are relatively direct. These perceptual distortions could serve to expand representational space and thereby enhance discriminability of gazes that are relatively direct. We also found that the emergent perception of gaze supports an abundance of direct gaze metamers-different combinations of head and pupil rotations that combine to generate the appearance of gaze directed toward the observer. Our results thus demonstrate a way in which the visual system flexibly integrates information from facial features to optimize social perception. Many gazes can be made to look toward you, yet similar gazes need not appear alike.

Asymmetry in Gaze Direction Discrimination Between the Upper and Lower Visual Fields

Previous research has shown that gaze direction can only be accurately discriminated within parafoveal limits (∼5° eccentricity) along the horizontal visual field. Beyond this eccentricity, head orientation seems to influence gaze discrimination more than iris cues. The present study examined gaze discrimination performance in the upper visual field (UVF) and lower visual field (LVF), and whether head orientation affects gaze judgments beyond parafoveal vision. Direct and averted gaze faces, in frontal and deviated head orientations, were presented for 150 ms along the vertical meridian while participants maintained central fixation during gaze discrimination judgments. Gaze discrimination was above chance level at all but one eccentricity for the two gaze-head congruent conditions. In contrast, for the incongruent conditions, gaze was discriminated above chance only from -1.5° to +3°, with an asymmetry between the UVF and LVF. Beyond foveal vision, response rates were biased toward head orientation rather than iris eccentricity, occurring in the LVF for both head orientations, and in the UVF for frontal head views. These findings suggest that covert processing of gaze direction involves the integration of eyes and head cues, with congruency of these two social cues driving response differences between the LVF and the UVF.

Spatial limitations in averaging social cues

The direction of social attention from groups provides stronger cueing than from an individual. It has previously been shown that both basic visual features such as size or orientation and more complex features such as face emotion and identity can be averaged across multiple elements. Here we used an equivalent noise procedure to compare observers' ability to average social cues with their averaging of a non-social cue. Estimates of observers' internal noise (uncertainty associated with processing any individual) and sample-size (the effective number of gaze-directions pooled) were derived by fitting equivalent noise functions to discrimination thresholds. We also used reverse correlation analysis to estimate the spatial distribution of samples used by participants. Averaging of head-rotation and cone-rotation was less noisy and more efficient than averaging of gaze direction, though presenting only the eye region of faces at a larger size improved gaze averaging performance. The reverse correlation analysis revealed greater sampling areas for head rotation compared to gaze. We attribute these differences in averaging between gaze and head cues to poorer visual processing of faces in the periphery. The similarity between head and cone averaging are examined within the framework of a general mechanism for averaging of object rotation.

The Effect of Head Orientation on Perceived Gaze Direction: Revisiting Gibson and Pick (1963) and Cline (1967)

Two biases in perceived gaze direction have been observed when eye and head orientation are not aligned. An overshoot effect indicates that perceived gaze direction is shifted away from head orientation (i.e., a repulsive effect), whereas a towing effect indicates that perceived gaze direction falls in between head and eye orientation (i.e., an attraction effect). In the 60s, three influential papers have been published with respect to the effect of head orientation on perceived gaze direction (Gibson and Pick, 1963; Cline, 1967; Anstis et al., 1969). Throughout the years, the results of two of these (Gibson and Pick, 1963; Cline, 1967) have been interpreted differently by a number of authors. In this paper, we critically discuss potential sources of confusion that have led to differential interpretations of both studies. At first sight, the results of Cline (1967), despite having been a major topic of discussion, unambiguously seem to indicate a towing effect whereas Gibson and Pick’s (1963) results seem to be the most ambiguous, although they have never been questioned in the literature. To shed further light on this apparent inconsistency, we repeated the critical experiments reported in both studies. Our results indicate an overshoot effect in both studies.

Automatic attentional orienting to other people's gaze in schizophrenia

Explicit tests of social cognition have revealed pervasive deficits in schizophrenia. Less is known of automatic social cognition in schizophrenia. We used a spatial orienting task to investigate automatic shifts of attention cued by another person's eye gaze in 29 patients and 28 controls. Central photographic images of a face with eyes shifted left or right, or looking straight ahead, preceded targets that appeared left or right of the cue. To examine automatic effects, cue direction was non-predictive of target location. Cue-target intervals were 100, 300, and 800 ms. In non-social control trials, arrows replaced eye-gaze cues. Both groups showed automatic attentional orienting indexed by faster reaction times (RTs) when arrows were congruent with target location across all cue-target intervals. Similar congruency effects were seen for eye-shift cues at 300 and 800 ms intervals, but patients showed significantly larger congruency effects at 800 ms, which were driven by delayed responses to incongruent target locations. At short 100-ms cue-target intervals, neither group showed faster RTs for congruent than for incongruent eye-shift cues, but patients were significantly slower to detect targets after direct-gaze cues. These findings conflict with previous studies using schematic line drawings of eye-shifts that have found automatic attentional orienting to be reduced in schizophrenia. Instead, our data indicate that patients display abnormalities in responding to gaze direction at various stages of gaze processing-reflected by a stronger preferential capture of attention by another person's direct eye contact at initial stages of gaze processing and difficulties disengaging from a gazed-at location once shared attention is established.

Perceiving where another person is looking: the integration of head and body information in estimating another person's gaze

The process through which an observer allocates his/her attention based on the attention of another person is known as joint attention. To be able to do this, the observer effectively has to compute where the other person is looking. It has been shown that observers integrate information from the head and the eyes to determine the gaze of another person. Most studies have documented that observers show a bias called the overshoot effect when eyes and head are misaligned. That is, when the head is not oriented straight to the observer, perceived gaze direction is sometimes shifted in the direction opposite to the head turn. The present study addresses whether body information is also used as a cue to compute perceived gaze direction. In Experiment 1, we observed a similar overshoot effect in both behavioral and saccadic responses when manipulating body orientation. In Experiment 2, we explored whether the overshoot effect was due to observers assuming that the eyes are oriented further than the head when head and body orientation are misaligned. We removed horizontal eye information by presenting the stimulus from a side view. Head orientation was now manipulated in a vertical direction and the overshoot effect was replicated. In summary, this study shows that body orientation is indeed used as a cue to determine where another person is looking.