Examining joint attention with the use of humanoid robots-A new approach to study fundamental mechanisms of social cognition

Tools and methods to study and replicate experiments addressing human social cognition in interactive scenarios

In the last decade, scientists investigating human social cognition have started bringing traditional laboratory paradigms more "into the wild" to examine how socio-cognitive mechanisms of the human brain work in real-life settings. As this implies transferring 2D observational paradigms to 3D interactive environments, there is a risk of compromising experimental control. In this context, we propose a methodological approach which uses humanoid robots as proxies of social interaction partners and embeds them in experimental protocols that adapt classical paradigms of cognitive psychology to interactive scenarios. This allows for a relatively high degree of "naturalness" of interaction and excellent experimental control at the same time. Here, we present two case studies where our methods and tools were applied and replicated across two different laboratories, namely the Italian Institute of Technology in Genova (Italy) and the Agency for Science, Technology and Research in Singapore. In the first case study, we present a replication of an interactive version of a gaze-cueing paradigm reported in Kompatsiari et al. (J Exp Psychol Gen 151(1):121-136, 2022). The second case study presents a replication of a "shared experience" paradigm reported in Marchesi et al. (Technol Mind Behav 3(3):11, 2022). As both studies replicate results across labs and different cultures, we argue that our methods allow for reliable and replicable setups, even though the protocols are complex and involve social interaction. We conclude that our approach can be of benefit to the research field of social cognition and grant higher replicability, for example, in cross-cultural comparisons of social cognition mechanisms.

Co-perceiving: Bringing the social into perception

Humans and other animals possess the remarkable ability to effectively navigate a shared perceptual environment by discerning which objects and spaces are perceived by others and which remain private to themselves. Traditionally, this capacity has been encapsulated under the umbrella of joint attention or joint action. In this comprehensive review, we advocate for a broader and more mechanistic understanding of this phenomenon, termed co-perception. Co-perception encompasses the sensitivity to the perceptual engagement of others and the capability to differentiate between objects perceived privately and those perceived commonly with others. It represents a distinct concept from mere simultaneous individual perception. Moreover, discerning between private and common objects doesn't necessitate intricate mind-reading abilities or mutual coordination. The act of perceiving objects as either private or common provides a comprehensive account for social scenarios where individuals simply share the same context or may even engage in competition. This conceptual framework encourages a re-examination of classical paradigms that demonstrate social influences on perception. Furthermore, it suggests that the impacts of shared experiences extend beyond affective responses, also influencing perceptual processes. This article is categorized under: Psychology > Attention Philosophy > Foundations of Cognitive Science Philosophy > Psychological Capacities.

Perceiving social gaze produces the reversed congruency effect

Numerous studies have shown that the gaze of others produces a special attentional process, such as the eye contact effect or joint attention. This study investigated the attentional process triggered by various types of gaze stimuli (i.e., human, cat, fish, koala, and robot gaze). A total of 300 university students participated in five experiments. They performed a spatial Stroop task in which five types of gaze stimuli were presented as targets. Participants were asked to judge the direction of the target (left or right) irrespective of its location (left or right). The results showed that the social gaze targets (i.e., human and cat gaze) produced a reversed congruency effect. In contrast to the social gaze targets, the non-social gaze (i.e., fish and robot) target did not produce the reversed congruency effect (Experiments 2, 2B, 3, and 4). These results suggest that attention to the gaze of socially communicable beings (i.e., humans and cats) is responsible for the reversed congruency effect. Our findings support the notion that the theory of mind or social interaction plays an important role in producing specific attentional processes in response to gaze stimuli.

Cultural differences in joint attention and engagement in mutual gaze with a robot face

Joint attention is a pivotal mechanism underlying human ability to interact with one another. The fundamental nature of joint attention in the context of social cognition has led researchers to develop tasks that address this mechanism and operationalize it in a laboratory setting, in the form of a gaze cueing paradigm. In the present study, we addressed the question of whether engaging in joint attention with a robot face is culture-specific. We adapted a classical gaze-cueing paradigm such that a robot avatar cued participants' gaze subsequent to either engaging participants in eye contact or not. Our critical question of interest was whether the gaze cueing effect (GCE) is stable across different cultures, especially if cognitive resources to exert top-down control are reduced. To achieve the latter, we introduced a mathematical stress task orthogonally to the gaze cueing protocol. Results showed larger GCE in the Singapore sample, relative to the Italian sample, independent of gaze type (eye contact vs. no eye contact) or amount of experienced stress, which translates to available cognitive resources. Moreover, since after each block, participants rated how engaged they felt with the robot avatar during the task, we observed that Italian participants rated as more engaging the avatar during the eye contact blocks, relative to no eye contact while Singaporean participants did not show any difference in engagement relative to the gaze. We discuss the results in terms of cultural differences in robot-induced joint attention, and engagement in eye contact, as well as the dissociation between implicit and explicit measures related to processing of gaze.

Can the robot "see" what I see? Robot gaze drives attention depending on mental state attribution

Mentalizing, where humans infer the mental states of others, facilitates understanding and interaction in social situations. Humans also tend to adopt mentalizing strategies when interacting with robotic agents. There is an ongoing debate about how inferred mental states affect gaze following, a key component of joint attention. Although the gaze from a robot induces gaze following, the impact of mental state attribution on robotic gaze following remains unclear. To address this question, we asked forty-nine young adults to perform a gaze cueing task during which mental state attribution was manipulated as follows. Participants sat facing a robot that turned its head to the screen at its left or right. Their task was to respond to targets that appeared either at the screen the robot gazed at or at the other screen. At the baseline, the robot was positioned so that participants would perceive it as being able to see the screens. We expected faster response times to targets at the screen the robot gazed at than targets at the non-gazed screen (i.e., gaze cueing effect). In the experimental condition, the robot's line of sight was occluded by a physical barrier such that participants would perceive it as unable to see the screens. Our results revealed gaze cueing effects in both conditions although the effect was reduced in the occluded condition compared to the baseline. These results add to the expanding fields of social cognition and human-robot interaction by suggesting that mentalizing has an impact on robotic gaze following.

Gazing left, gazing right: exploring a spatial bias in social attention

Faces oriented rightwards are sometimes perceived as more dominant than faces oriented leftwards. In this study, we explored whether faces oriented rightwards can also elicit increased attentional orienting. Participants completed a discrimination task in which they were asked to discriminate, by means of a keypress, a peripheral target. At the same time, a task-irrelevant face oriented leftwards or rightwards appeared at the centre of the screen. The results showed that, while for faces oriented rightwards targets appearing on the right were responded to faster as compared to targets appearing on the left, for faces oriented leftwards no differences emerged between left and right targets. Furthermore, we also found a negative correlation between the magnitude of the orienting response elicited by the faces oriented leftwards and the level of conservatism of the participants. Overall, these findings provide evidence for the existence of a spatial bias reflected in social orienting.

Age-Related Differences in the Perception of Robotic Referential Gaze in Human-Robot Interaction

There is an increased interest in using social robots to assist older adults during their daily life activities. As social robots are designed to interact with older users, it becomes relevant to study these interactions under the lens of social cognition. Gaze following, the social ability to infer where other people are looking at, deteriorates with older age. Therefore, the referential gaze from robots might not be an effective social cue to indicate spatial locations to older users. In this study, we explored the performance of older adults, middle-aged adults, and younger controls in a task assisted by the referential gaze of a Pepper robot. We examined age-related differences in task performance, and in self-reported social perception of the robot. Our main findings show that referential gaze from a robot benefited task performance, although the magnitude of this facilitation was lower for older participants. Moreover, perceived anthropomorphism of the robot varied less as a result of its referential gaze in older adults. This research supports that social robots, even if limited in their gazing capabilities, can be effectively perceived as social entities. Additionally, this research suggests that robotic social cues, usually validated with young participants, might be less optimal signs for older adults.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12369-022-00926-6.

Humans Can't Resist Robot Eyes - Reflexive Cueing With Pseudo-Social Stimuli

Joint attention is a key mechanism for humans to coordinate their social behavior. Whether and how this mechanism can benefit the interaction with pseudo-social partners such as robots is not well understood. To investigate the potential use of robot eyes as pseudo-social cues that ease attentional shifts we conducted an online study using a modified spatial cueing paradigm. The cue was either a non-social (arrow), a pseudo-social (two versions of an abstract robot eye), or a social stimulus (photographed human eyes) that was presented either paired (e.g. two eyes) or single (e.g. one eye). The latter was varied to separate two assumed triggers of joint attention: the social nature of the stimulus, and the additional spatial information that is conveyed only by paired stimuli. Results support the assumption that pseudo-social stimuli, in our case abstract robot eyes, have the potential to facilitate human-robot interaction as they trigger reflexive cueing. To our surprise, actual social cues did not evoke reflexive shifts in attention. We suspect that the robot eyes elicited the desired effects because they were human-like enough while at the same time being much easier to perceive than human eyes, due to a design with strong contrasts and clean lines. Moreover, results indicate that for reflexive cueing it does not seem to make a difference if the stimulus is presented single or paired. This might be a first indicator that joint attention depends rather on the stimulus' social nature or familiarity than its spatial expressiveness. Overall, the study suggests that using paired abstract robot eyes might be a good design practice for fostering a positive perception of a robot and to facilitate joint attention as a precursor for coordinated behavior.

Visual and Hearing Sensitivity Affect Robot-Based Training for Children Diagnosed With Autism Spectrum Disorder

In this paper, we investigate the impact of sensory sensitivity during robot-assisted training for children diagnosed with Autism Spectrum Disorder (ASD). Indeed, user-adaptation for robot-based therapies could help users to focus on the training, and thus improve the benefits of the interactions. Children diagnosed with ASD often suffer from sensory sensitivity, and can show hyper or hypo-reactivity to sensory events, such as reacting strongly or not at all to sounds, movements, or touch. Considering it during robot therapies may improve the overall interaction. In the present study, thirty-four children diagnosed with ASD underwent a joint attention training with the robot Cozmo. The eight session training was embedded in the standard therapy. The children were screened for their sensory sensitivity with the Sensory Profile Checklist Revised. Their social skills were screened before and after the training with the Early Social Communication Scale. We recorded their performance and the amount of feedback they were receiving from the therapist through animations of happy and sad emotions played on the robot. Our results showed that visual and hearing sensitivity influenced the improvements of the skill to initiate joint attention. Also, the therapists of individuals with a high sensitivity to hearing chose to play fewer animations of the robot during the training phase of the robot activity. The animations did not include sounds, but the robot was producing motor noise. These results are supporting the idea that sensory sensitivity of children diagnosed with ASD should be screened prior to engaging the children in robot-assisted therapy.

Irrelevant Robot Signals in a Categorization Task Induce Cognitive Conflict in Performance, Eye Trajectories, the N2 ERP-EEG Component, and Frontal Theta Oscillations

Understanding others' nonverbal behavior is essential for social interaction, as it allows, among others, to infer mental states. Although gaze communication, a well-established nonverbal social behavior, has shown its importance in inferring others' mental states, not much is known about the effects of irrelevant gaze signals on cognitive conflict markers during collaborative settings. Here, participants completed a categorization task where they categorized objects based on their color while observing images of a robot. On each trial, participants observed the robot iCub grasping an object from a table and offering it to them to simulate a handover. Once the robot "moved" the object forward, participants were asked to categorize the object according to its color. Before participants were allowed to respond, the robot made a lateral head/gaze shift. The gaze shifts were either congruent or incongruent with the object's color. We expected that incongruent head cues would induce more errors (Study 1), would be associated with more curvature in eye-tracking trajectories (Study 2), and induce larger amplitude in electrophysiological markers of cognitive conflict (Study 3). Results of the three studies show more oculomotor interference as measured in error rates (Study 1), larger curvatures eye-tracking trajectories (Study 2), and higher amplitudes of the N2 ERP of the EEG signals as well as higher event-related spectral perturbation amplitudes (Study 3) for incongruent trials compared with congruent trials. Our findings reveal that behavioral, ocular, and electrophysiological markers can index the influence of irrelevant signals during goal-oriented tasks.

Gaze aversion in conversational settings: An investigation based on mock job interview

We report the results of an empirical study on gaze aversion during dyadic human-to-human conversation in an interview setting. To address various methodological challenges in assessing gaze-to-face contact, we followed an approach where the experiment was conducted twice, each time with a different set of interviewees. In one of them the interviewer’s gaze was tracked with an eye tracker, and in the other the interviewee’s gaze was tracked. The gaze sequences obtained in both experiments were analyzed and modeled as Discrete-Time Markov Chains. The results show that the interviewer made more frequent and longer gaze contacts compared to the interviewee. Also, the interviewer made mostly diagonal gaze aversions, whereas the interviewee made sideways aversions (left or right). We discuss the relevance of this research for Human-Robot Interaction, and discuss some future research problems.

Eye contact during joint attention with a humanoid robot modulates oscillatory brain activity

Eye contact established by a human partner has been shown to affect various cognitive processes of the receiver. However, little is known about humans' responses to eye contact established by a humanoid robot. Here, we aimed at examining humans' oscillatory brain response to eye contact with a humanoid robot. Eye contact (or lack thereof) was embedded in a gaze-cueing task and preceded the phase of gaze-related attentional orienting. In addition to examining the effect of eye contact on the recipient, we also tested its impact on gaze-cueing effects (GCEs). Results showed that participants rated eye contact as more engaging and responded with higher desynchronization of alpha-band activity in left fronto-central and central electrode clusters when the robot established eye contact with them, compared to no eye contact condition. However, eye contact did not modulate GCEs. The results are interpreted in terms of the functional roles involved in alpha central rhythms (potentially interpretable also as mu rhythm), including joint attention and engagement in social interaction.

What Makes a Robot Social? A Review of Social Robots from Science Fiction to a Home or Hospital Near You

Purpose of Review

We provide an outlook on the definitions, laboratory research, and applications of social robots, with an aim to understand what makes a robot social—in the eyes of science and the general public.

Recent Findings

Social robots demonstrate their potential when deployed within contexts appropriate to their form and functions. Some examples include companions for the elderly and cognitively impaired individuals, robots within educational settings, and as tools to support cognitive and behavioural change interventions.

Summary

Science fiction has inspired us to conceive of a future with autonomous robots helping with every aspect of our daily lives, although the robots we are familiar with through film and literature remain a vision of the distant future. While there are still miles to go before robots become a regular feature within our social spaces, rapid progress in social robotics research, aided by the social sciences, is helping to move us closer to this reality.

Social Robots to Test Flexibility of Human Social Cognition

As the field of social robotics has been dynamically growing and expanding over various areas of research and application, in which robots can be of assistance and companionship for humans, this paper offers a different perspective on a role that social robots can also play, namely the role of informing us about flexibility of human mechanisms of social cognition. The paper focuses on studies in which robots have been used as a new type of "stimuli" in psychological experiments to examine whether similar mechanisms of social cognition would be activated in interaction with a robot, as would be elicited in interaction with another human. Analysing studies in which a direct comparison has been made between a robot and a human agent, the paper examines whether for robot agents, the brain re-uses the same mechanisms that have been developed for interaction with other humans in terms of perception, action representation, attention and higher-order social cognition. Based on this analysis, the paper concludes that the human socio-cognitive mechanisms, in adult brains, are sufficiently flexible to be re-used for robotic agents, at least for those that have some level of resemblance to humans.