The dual nature of eye contact: to see and to be seen

Psychophysiological responses to eye contact with a humanoid robot: Impact of perceived intentionality

Eye contact with a social robot has been shown to elicit similar psychophysiological responses to eye contact with another human. However, it is becoming increasingly clear that the attention- and affect-related psychophysiological responses differentiate between direct (toward the observer) and averted gaze mainly when viewing embodied faces that are capable of social interaction, whereas pictorial or pre-recorded stimuli have no such capability. It has been suggested that genuine eye contact, as indicated by the differential psychophysiological responses to direct and averted gaze, requires a feeling of being watched by another mind. Therefore, we measured event-related potentials (N170 and frontal P300) with EEG, facial electromyography, skin conductance, and heart rate deceleration responses to seeing a humanoid robot's direct versus averted gaze, while manipulating the impression of the robot's intentionality. The results showed that the N170 and the facial zygomatic responses were greater to direct than to averted gaze of the robot, and independent of the robot's intentionality, whereas the frontal P300 responses were more positive to direct than to averted gaze only when the robot appeared intentional. The study provides further evidence that the gaze behavior of a social robot elicits attentional and affective responses and adds that the robot's seemingly autonomous social behavior plays an important role in eliciting higher-level socio-cognitive processing.

Depth from motion parallax: Deictic consistency, eye contact, and a serious problem with Zoom

The dynamics of head and eye gaze between two or more individuals displayed during verbal and nonverbal face-to-face communication contains a wealth of information and is used for both volitionary and unconscious signaling. Current video communication systems convey visual signals about gaze behavior and other directional cues, but the information they carry is often spurious and potentially misleading. I discuss the consequences of this situation, identify the source of the problem as a more general lack of deictic consistency, and demonstrate that using display technologies that simulate motion parallax are both necessary and sufficient to alleviate it. I then devise an avatar-based remote communication solution that achieves deictic consistency and provides natural, dynamic eye contact for computer-mediated audiovisual communication.

I see you, you see me: the impact of social presence on social interaction processes in autistic and non-autistic people

Environments that require social interaction are complex, challenging and sometimes experienced as overwhelming by autistic people. However, all too often theories relating to social interaction processes are created, and interventions are proposed, on the basis of data collected from studies that do not involve genuine social encounters nor do they consider the perception of social presence to be a potentially influential factor. In this review, we begin by considering why face-to-face interaction research is important in this field. We then discuss how the perception of social agency and social presence can influence conclusions about social interaction processes. We then outline some insights gained from face-to-face interaction research conducted with both autistic and non-autistic people. We finish by considering the impact of social presence on cognitive processes more broadly, including theory of mind. Overall, we demonstrate that choice of stimuli in studies assessing social interaction processes has the potential to substantially alter conclusions drawn. Ecological validity matters and social presence, in particular, is a critical factor that fundamentally impacts social interaction processes in both autistic and non-autistic people. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Research on social attention in autism and the challenges of the research domain criteria (RDoC) framework

The fuzzy nature of categories of psychopathology, such as autism, leads to significant research challenges. Alternatively, focusing research on the study of a common set of important and well-defined psychological constructs across psychiatric conditions may make the fundamental etiological processes of psychopathology easier to discern and treat (Cuthbert, 2022). The development of the research domain criteria (RDoC) framework is designed to guide this new research approach (Insel et al., 2010). However, progress in research may be expected to continually refine and reorganize the understanding of the specifics of these mental processes (Cuthbert & Insel, 2013). Moreover, knowledge gleaned from the study of both normative and atypical development can be mutually informative in the evolution of our understanding of these fundamental processes. A case in point is the study of social attention. This Autism 101 commentary provides an educational summary of research over the last few decades indicates that social attention is major construct in the study of human social-cognitive development, autism and other forms of psychopathology. The commentary also describes how this research can inform the Social Process dimension of the RDoC framework.

Skin conductance, facial EMG, and heart rate responses in multi-person gaze interactions

Previous literature has reported enhanced affective and attentional responses to faces with a direct vs. averted gaze. Typically, in these studies, only single faces were presented. However, daily social encounters often involve interaction with more than just one person. By employing an experimental set-up in which the participants believed they were interacting with two other persons, the present study, for the first time, investigated participants' skin conductance, facial electromyographic (EMG), and heart rate deceleration responses in multi-person eye contact situations. Responses were measured in two different social contexts; i) when the participants observed eye contact between two other persons ('vicarious eye contact effect'), and ii) when the participants themselves received direct gaze either from one or two persons. The results showed that the skin conductance, facial EMG, and heart rate deceleration responses elicited by observing two other persons making eye contact did not differ from those elicited by observing one person looking at the other while the other person was not reciprocating with their gaze. As a novel finding, the results showed that receiving direct gaze from two persons elicited greater affective arousal and zygomatic EMG, but smaller heart rate deceleration responses in participants than receiving direct gaze from one person only. The findings are thoroughly discussed and it is concluded that physiological responses in multi-person interaction contexts are influenced by many social effects between the interactors and can be markedly different from those observed in two-person interactions.

The Unique Cost of Human Eye Gaze in Cognitive Control: Being Human-Specific and Body-Related?

This study investigated the eye gaze cost in cognitive control and whether it is human-specific and body-related. In Experiment 1, we explored whether there was a cost of human eye gaze in cognitive control and extended it by focusing on the role of emotion in the cost. Stroop effect was found to be larger in eye-gaze condition than vertical grating condition, and to be comparable across positive, negative, and neutral trials. In Experiment 2, we explored whether the eye gaze cost in cognitive control was limited to human eyes. No larger Stroop effect was found in feline eye-gaze condition, neither the modulating role of emotion. In Experiment 3, we explored whether the mouth could elicit a cost in Stroop effect. Stroop effect was not significantly larger in mouth condition compared to vertical grating condition, nor across positive, negative, and neutral conditions. The results suggest that: (1) There is a robust cost of eye gaze in cognitive control; (2) Such eye-gaze cost was specific to human eyes but not to animal eyes; (3) Only human eyes could have such eye-gaze costs but not human mouth. This study supported the notion that presentation of social cues, such as human eyes, could influence attentional processing, and provided preliminary evidence that the human eye plays an important role in cognitive processing.

Communicative roots of complex sociality and cognition: neuropsychological mechanisms underpinning the processing of social information

Primate social bonds are described as being especially complex in their nature, and primates have unusually large brains for their body size compared to other mammals. Communication in primates has attracted considerable attention because of the important role it plays in social bonding. It has been proposed that differentiated social relationships are cognitively complex because primates need to continuously update their knowledge about different types of social bonds. Therefore, primates infer whether an opportunity for social interaction is rewarding (valuable to individual goals) based on their knowledge of the social relationships of the interactants. However, exposure to distraction and stress has detrimental effects on the dopaminergic system, suggesting that understanding social relationships as rewarding is affected in these conditions. This paper proposes that complex communication evolved to augment the capacity to form social relationships during stress through flexibly modifying intentionality in communication (audience checking, response waiting and elaboration). Intentional communication may upregulate dopamine dynamics to allow recognition that an interaction is rewarding during stress. By examining these associations between complexity of communication and stress, we provide new insights into the cognitive skills involved in forming social bonds in primates and the evolution of communication systems in both primates and humans. This article is part of the theme issue 'Cognition, communication and social bonds in primates'.

Attention to live eye contact in adolescents with autism spectrum disorder

A number of studies have reported diminished attention to the eyes in individuals with autism spectrum disorder (ASD). These studies predominantly used static images of faces as stimuli. Recent studies, however, have shown enhanced response to eye contact in typically developing (TD) individuals when they observe a person in a live interaction. We investigated physiological orienting to perceived eye contact in adolescents with ASD and TD adolescents when they observed a person in live interaction or viewed a photograph of the same person's face. We measured heart rate (HR) deceleration as an index of attentional orienting. Adolescents with ASD, as well as TD adolescents, showed significant HR deceleration for the direct gaze compared to an averted gaze in the live condition, but not in the photographic condition. The results suggest an intact response to perceived eye contact in individuals with ASD during a live face-to-face interaction. LAY SUMMARY: Individuals with autism spectrum disorder (ASD) have a different eye gaze pattern when observing photographic faces. However, little is known about how individuals with ASD process a real person's face. We measured heart rate (HR) and found that adolescents with ASD showed the typical decline in HR when they made eye contact with a real person, which suggests that both groups of individuals directed their attention to eye contact in a live face-to-face interaction.

From Gaze Perception to Social Cognition: The Shared-Attention System

When two people look at the same object in the environment and are aware of each other's attentional state, they find themselves in a shared-attention episode. This can occur through intentional or incidental signaling and, in either case, causes an exchange of information between the two parties about the environment and each other's mental states. In this article, we give an overview of what is known about the building blocks of shared attention (gaze perception and joint attention) and focus on bringing to bear new findings on the initiation of shared attention that complement knowledge about gaze following and incorporate new insights from research into the sense of agency. We also present a neurocognitive model, incorporating first-, second-, and third-order social cognitive processes (the shared-attention system, or SAS), building on previous models and approaches. The SAS model aims to encompass perceptual, cognitive, and affective processes that contribute to and follow on from the establishment of shared attention. These processes include fundamental components of social cognition such as reward, affective evaluation, agency, empathy, and theory of mind.

Why students do not turn on their video cameras during online classes and an equitable and inclusive plan to encourage them to do so

Enrollment in courses taught remotely in higher education has been on the rise, with a recent surge in response to a global pandemic. While adapting this form of teaching, instructors familiar with traditional face-to-face methods are now met with a new set of challenges, including students not turning on their cameras during synchronous class meetings held via videoconferencing. After transitioning to emergency remote instruction in response to the COVID-19 pandemic, our introductory biology course shifted all in-person laboratory sections into synchronous class meetings held via the Zoom videoconferencing program. Out of consideration for students, we established a policy that video camera use during class was optional, but encouraged. However, by the end of the semester, several of our instructors and students reported lower than desired camera use that diminished the educational experience. We surveyed students to better understand why they did not turn on their cameras. We confirmed several predicted reasons including the most frequently reported: being concerned about personal appearance. Other reasons included being concerned about other people and the physical location being seen in the background and having a weak internet connection, all of which our exploratory analyses suggest may disproportionately influence underrepresented minorities. Additionally, some students revealed to us that social norms also play a role in camera use. This information was used to develop strategies to encourage-without requiring-camera use while promoting equity and inclusion. Broadly, these strategies are to not require camera use, explicitly encourage usage while establishing norms, address potential distractions, engage students with active learning, and understand your students' challenges through surveys. While the demographics and needs of students vary by course and institution, our recommendations will likely be directly helpful to many instructors and also serve as a model for gathering data to develop strategies more tailored for other student populations.

The N170 event-related potential differentiates congruent and incongruent gaze responses in gaze leading

To facilitate social interactions, humans need to process the responses that other people make to their actions, including eye movements that could establish joint attention. Here, we investigated the neurophysiological correlates of the processing of observed gaze responses following the participants’ own eye movement. These observed gaze responses could either establish, or fail to establish, joint attention. We implemented a gaze leading paradigm in which participants made a saccade from an on-screen face to an object, followed by the on-screen face either making a congruent or incongruent gaze shift. An N170 event-related potential was elicited by the peripherally located gaze shift stimulus. Critically, the N170 was greater for joint attention than non-joint gaze both when task-irrelevant (Experiment 1) and task-relevant (Experiment 2). These data suggest for the first time that the neurocognitive system responsible for structural encoding of face stimuli is affected by the establishment of participant-initiated joint attention.

Psychophysiological responses to eye contact in a live interaction and in video call

Another person's gaze directed to oneself elicits autonomic arousal and facial reactions indicating positive affect in its observer. These effects have only been found to occur with mutual, live eye contact and not in response to direct gaze pictures or when the observer believes that the live person cannot see them. The question remains whether the physical presence of the other person is necessary for these effects. We measured psychophysiological responses to another person's direct versus averted gaze in three conditions: live interaction, bidirectional video call, and watching a mere video. Autonomic arousal was measured with skin conductance responses and facial reactions with facial electromyography. In the live and video call conditions, but not in the mere video condition, direct gaze increased autonomic arousal in comparison to averted gaze. In all three conditions, however, direct gaze elicited positive affective facial reactions. Therefore, an experience of being seen is essential for the autonomic reactions but not for the facial responses that are elicited by another person's direct gaze. Most importantly, the results suggest that the physical presence or proximity of the other person is not necessary for these psychophysiological responses to eye contact.

Eye Contact Is a Two-Way Street: Arousal Is Elicited by the Sending and Receiving of Eye Gaze Information

Research shows that arousal is significantly enhanced while participants make eye contact with a live person compared to viewing a picture of direct or averted gaze. Recent research has pointed toward the potential for social interaction as a possible driving force behind the arousal enhancement. That is, eye gaze is not only a signal perceived but also a signal sent out in order to communicate with others. This study aimed to test this by having dyads engage in eye contact and averted gaze naturally, while wearing sunglasses, and while blindfolded; such that the gaze signals were clear, degraded, and blocked, respectively. Autonomic nervous system arousal was measured via skin conductance response and level. The results showed that dyads exhibited the highest degree of arousal (increased skin conductance) while making eye contact (send/receive) compared to send-only or receive-only gaze trials; however, this was only the case if eye contact was clear. Once gaze information became degraded (by sunglasses or blindfold), arousal significantly decreased and was no longer modulated by the sending and receiving of gaze. Therefore, the arousal enhancement observed during eye contact is not only caused by receiving gaze signals (the focus of previous research) and should be more accurately attributed to the subtle interplay between sending and receiving gaze signals.

Socially Communicative Eye Contact and Gender Affect Memory

Because of their value as a socially communicative cue, researchers have strived to understand how the gaze of other people influences a variety of cognitive processes. Recent work in social attention suggests that the use of images of people in laboratory studies, as a substitute for real people, may not effectively test socially communicative aspects of eye gaze. As attention affects many other cognitive processes, it is likely that social attention between real individuals could also affect other cognitive processes, such as memory. However, from previous work alone, it is unclear whether, and if so how, socially communicative eye gaze affects memory. The present studies test the assumption that socially communicative aspects of eye gaze may impact memory by manipulating the eye gaze of a live speaker in the context of a traditional recognition paradigm used frequently in the laboratory. A female (Experiment 1) or male (Experiment 2) investigator read words aloud and varied whether eye contact was, or was not, made with a participant. With both female and male investigators, eye contact improved word recognition only for female participants and hindered word recognition in male participants. When a female investigator prolonged their eye contact (Experiment 3) to provide a longer opportunity to both observe and process the investigator's eye gaze, the results replicated the findings from Experiments 1 and 2. The findings from Experiments 1-3 suggest that females interpret and use the investigator's eye gaze differently than males. When key aspects from the previous experiments were replicated in a noncommunicative situation (i.e., when a video of a speaker is used instead of a live speaker; Experiment 4), the memory effects observed previously in response to eye gaze were eliminated. Together, these studies suggest that it is the socially communicative aspects of eye gaze from a real person that influence memory. The findings reveal the importance of using social cues that are communicative in nature (e.g., real people) when studying the relationship between social attention and memory.

The Role of Eye Gaze During Natural Social Interactions in Typical and Autistic People

Social interactions involve complex exchanges of a variety of social signals, such as gaze, facial expressions, speech and gestures. Focusing on the dual function of eye gaze, this review explores how the presence of an audience, communicative purpose and temporal dynamics of gaze allow interacting partners to achieve successful communication. First, we focus on how being watched modulates social cognition and behavior. We then show that the study of interpersonal gaze processing, particularly gaze temporal dynamics, can provide valuable understanding of social behavior in real interactions. We propose that the Interpersonal Gaze Processing model, which combines both sensing and signaling functions of eye gaze, provides a framework to make sense of gaze patterns in live interactions. Finally, we discuss how autistic individuals process the belief in being watched and interpersonal dynamics of gaze, and suggest that systematic manipulation of factors modulating gaze signaling can reveal which aspects of social eye gaze are challenging in autism.

Affective Eye Contact: An Integrative Review

In recent years, many studies have shown that perceiving other individuals' direct gaze has robust effects on various attentional and cognitive processes. However, considerably less attention has been devoted to investigating the affective effects triggered by eye contact. This article reviews research concerning the effects of others' gaze direction on observers' affective responses. The review focuses on studies in which affective reactions have been investigated in well-controlled laboratory experiments, and in which contextual factors possibly influencing perceivers' affects have been controlled. Two important themes emerged from this review. First, explicit affective evaluations of seeing another's direct versus averted gaze have resulted in rather inconsistent findings; some studies report more positive subjective feelings to direct compared to averted gaze, whereas others report the opposite pattern. These contradictory findings may be related, for example, to differences between studies in terms of the capability of direct-gaze stimuli to elicit feelings of self-involvement. Second, studies relying on various implicit measures have reported more consistent results; they indicate that direct gaze increases affective arousal, and more importantly, that eye contact automatically evokes a positively valenced affective reaction. Based on the review, possible psychological mechanisms for the positive affective reactions elicited by eye contact are described.

How attention gates social interactions

Social interactions are at the core of social life. However, humans selectively choose their exchange partners and do not engage in all available opportunities for social encounters. In this review, we argue that attentional systems play an important role in guiding the selection of social interactions. Supported by both classic and emerging literature, we identify and characterize the three core processes-perception, interpretation, and evaluation-that interact with attentional systems to modulate selective responses to social environments. Perceptual processes facilitate attentional prioritization of social cues. Interpretative processes link attention with understanding of cues' social meanings and agents' mental states. Evaluative processes determine the perceived value of the source of social information. The interplay between attention and these three routes of processing places attention in a powerful role to manage the selection of the vast amount of social information that individuals encounter on a daily basis and, in turn, gate the selection of social interactions.

Using dual eye tracking to uncover personal gaze patterns during social interaction

We report the personal eye gaze patterns of people engaged in face-to-face getting acquainted conversation. Considerable differences between individuals are underscored by a stability of eye gaze patterns within individuals. Results suggest the existence of an eye-mouth gaze continuum. This continuum includes some people showing a strong preference for eye gaze, some with a strong preference for mouth gaze, and others distributing their gaze between the eyes and mouth to varying extents. Additionally, we found evidence of within-participant consistency not just for location preference but also for the duration of fixations upon the eye and mouth regions. We also estimate that during a 4-minute getting acquainted conversation mutual face gaze constitutes about 60% of conversation that occurs via typically brief instances of 2.2 seconds. Mutual eye contact ranged from 0-45% of conversation, via very brief instances. This was despite participants subjectively perceiving eye contact occurring for about 70% of conversation. We argue that the subjective perception of eye contact is a product of mutual face gaze instead of actual mutual eye contact. We also outline the fast activity of gaze movements upon various locations both on and off face during a typical face-to-face conversation.

Frontal temporal and parietal systems synchronize within and across brains during live eye-to-eye contact

Human eye-to-eye contact is a primary source of social cues and communication. In spite of the biological significance of this interpersonal interaction, the underlying neural processes are not well-understood. This knowledge gap, in part, reflects limitations of conventional neuroimaging methods, including solitary confinement in the bore of a scanner and minimal tolerance of head movement that constrain investigations of natural, two-person interactions. However, these limitations are substantially resolved by recent technical developments in functional near-infrared spectroscopy (fNIRS), a non-invasive spectral absorbance technique that detects changes in blood oxygen levels in the brain by using surface-mounted optical sensors. Functional NIRS is tolerant of limited head motion and enables simultaneous acquisitions of neural signals from two interacting partners in natural conditions. We employ fNIRS to advance a data-driven theoretical framework for two-person neuroscience motivated by the Interactive Brain Hypothesis which proposes that interpersonal interaction between individuals evokes neural mechanisms not engaged during solo, non-interactive, behaviors. Within this context, two specific hypotheses related to eye-to-eye contact, functional specificity and functional synchrony, were tested. The functional specificity hypothesis proposes that eye-to-eye contact engages specialized, within-brain, neural systems; and the functional synchrony hypothesis proposes that eye-to-eye contact engages specialized, across-brain, neural processors that are synchronized between dyads. Signals acquired during eye-to-eye contact between partners (interactive condition) were compared to signals acquired during mutual gaze at the eyes of a picture-face (non-interactive condition). In accordance with the specificity hypothesis, responses during eye-to-eye contact were greater than eye-to-picture gaze for a left frontal cluster that included pars opercularis (associated with canonical language production functions known as Broca's region), pre- and supplementary motor cortices (associated with articulatory systems), as well as the subcentral area. This frontal cluster was also functionally connected to a cluster located in the left superior temporal gyrus (associated with canonical language receptive functions known as Wernicke's region), primary somatosensory cortex, and the subcentral area. In accordance with the functional synchrony hypothesis, cross-brain coherence during eye-to-eye contact relative to eye-to-picture gaze increased for signals originating within left superior temporal, middle temporal, and supramarginal gyri as well as the pre- and supplementary motor cortices of both interacting brains. These synchronous cross-brain regions are also associated with known language functions, and were partner-specific (i.e., disappeared with randomly assigned partners). Together, both within and across-brain neural correlates of eye-to-eye contact included components of previously established productive and receptive language systems. These findings reveal a left frontal, temporal, and parietal long-range network that mediates neural responses during eye-to-eye contact between dyads, and advance insight into elemental mechanisms of social and interpersonal interactions.

Audience effects: what can they tell us about social neuroscience, theory of mind and autism?

An audience effect arises when a person’s behaviour changes because they believe someone else is watching them. Though these effects have been known about for over 110 years, the cognitive mechanisms of the audience effect and how it might vary across different populations and cultures remains unclear. In this review, we examine the hypothesis that the audience effect draws on implicit mentalising abilities. Behavioural and neuroimaging data from a number of tasks are consistent with this hypothesis. We further review data suggest that how people respond to audiences may vary over development, personality factors, cultural background and clinical diagnosis including autism and anxiety disorder. Overall, understanding and exploring the audience effect may contribute to our models of social interaction, including reputation management and mentalising.

The duality of gaze: eyes extract and signal social information during sustained cooperative and competitive dyadic gaze

In contrast to non-human primate eyes, which have a dark sclera surrounding a dark iris, human eyes have a white sclera that surrounds a dark iris. This high contrast morphology allows humans to determine quickly and easily where others are looking and infer what they are attending to. In recent years an enormous body of work has used photos and schematic images of faces to study these aspects of social attention, e.g., the selection of the eyes of others and the shift of attention to where those eyes are directed. However, evolutionary theory holds that humans did not develop a high contrast morphology simply to use the eyes of others as attentional cues; rather they sacrificed camouflage for communication, that is, to signal their thoughts and intentions to others. In the present study we demonstrate the importance of this by taking as our starting point the hypothesis that a cornerstone of non-verbal communication is the eye contact between individuals and the time that it is held. In a single simple study we show experimentally that the effect of eye contact can be quickly and profoundly altered merely by having participants, who had never met before, play a game in a cooperative or competitive manner. After the game participants were asked to make eye contact for a prolonged period of time (10 min). Those who had played the game cooperatively found this terribly difficult to do, repeatedly talking and breaking gaze. In contrast, those who had played the game competitively were able to stare quietly at each other for a sustained period. Collectively these data demonstrate that when looking at the eyes of a real person one both acquires and signals information to the other person. This duality of gaze is critical to non-verbal communication, with the nature of that communication shaped by the relationship between individuals, e.g., cooperative or competitive.