Testing the relationship between mimicry, trust and rapport in virtual reality conversations

The effect of non-verbal mimicry on evaluations in interactions with cognitively (dis)similar individuals

Non-verbal mimicry (i.e., being posturally similar by copying another person's body language) has been shown to increase evaluations of the mimicker. Concurrently, extensive research in social psychology has demonstrated a negative effect on interpersonal evaluations when one perceives others as cognitively dissimilar, often resulting in interpersonal conflicts. Across two experiments (Experiment 1: N = 159, Experiment 2: N = 144), we tested our hypotheses that mimicry, compared with no mimicry, will make mimickers come across as more likable and competent regardless of whether they were perceived as cognitively dissimilar or not (Experiment 1) and regardless of the extent to which they were perceived as cognitively dissimilar (Experiment 2). Broadly, we found support for our hypotheses, and via mediation sensitivity analyses, we found that the effect of mimicry, at least for likability, was mediated by participants' perceived personal similarity to the mimicker. Non-verbal mimicry may thus be one way of alleviating interpersonal conflicts via increasing perceptions of personal similarity regardless of initial cognitive dissimilarity.

A randomised controlled test in virtual reality of the effects on paranoid thoughts of virtual humans' facial animation and expression

Virtual reality (VR) is increasingly used in the study and treatment of paranoia. This is based on the finding that people who mistakenly perceive hostile intent from other people also perceive similar threat from virtual characters. However, there has been no study of the programming characteristics of virtual characters that may influence their interpretation. We set out to investigate how the animation and expressions of virtual humans may affect paranoia. In a two-by-two factor, between-groups, randomized design, 122 individuals with elevated paranoia rated their perceptions of virtual humans, set in an eye-tracking enabled VR lift scenario, that varied in facial animation (static or animated) and expression (neutral or positive). Both facial animation (group difference = 102.328 [51.783, 152.872], p < 0.001,η p 2 = 0.125) and positive expressions (group difference = 53.016 [0.054, 105.979], p = 0.049,η p 2 = 0.033) led to less triggering of paranoid thoughts about the virtual humans. Facial animation (group difference = 2.442 [- 4.161, - 0.724], p = 0.006,η p 2 = 0.063) but not positive expressions (group difference = 0.344 [- 1.429, 2.110], p = 0.681,η p 2 = 0.001) significantly increased the likelihood of neutral thoughts about the characters. Our study shows that the detailed programming of virtual humans can impact the occurrence of paranoid thoughts in VR. The programming of virtual humans needs careful consideration depending on the purpose of their use.

Sensorimotor regulation of facial expression - An untouched frontier

Facial expression is a critical form of nonverbal social communication which promotes emotional exchange and affiliation among humans. Facial expressions are generated via precise contraction of the facial muscles, guided by sensory feedback. While the neural pathways underlying facial motor control are well characterized in humans and primates, it remains unknown how tactile and proprioceptive information reaches these pathways to guide facial muscle contraction. Thus, despite the importance of facial expressions for social functioning, little is known about how they are generated as a unique sensorimotor behavior. In this review, we highlight current knowledge about sensory feedback from the face and how it is distinct from other body regions. We describe connectivity between the facial sensory and motor brain systems, and call attention to the other brain systems which influence facial expression behavior, including vision, gustation, emotion, and interoception. Finally, we petition for more research on the sensory basis of facial expressions, asserting that incomplete understanding of sensorimotor mechanisms is a barrier to addressing atypical facial expressivity in clinical populations.

The subjective value of genuine smiles guides real-world social behaviour

The mechanisms that underpin human social behaviour are poorly understood, in part because natural social behaviour is challenging to study. The task of linking the mechanisms thought to drive social behaviour to specific social behaviours in a manner that maintains ecological validity poses an even greater challenge. Here we report evidence that the subjective value people assign to genuine smiles, as measured in the laboratory, determines their responsiveness to genuine smiles encountered in a naturalistic social interaction. Specifically, participants (university undergraduates; age 17 to 36) who valued genuine smiles to a greater degree also showed stronger attention capture effects to neutral faces that were previously associated with genuine smiles and faster reciprocity of a social partner's smiles in a real social interaction. Additionally, the faster participants responded to the partner's genuine smiles the higher the partner's ratings of interaction quality were after the interaction. These data suggest that individual differences in subjective value of genuine smiles, measured in the lab, is one element that underpins responsiveness to natural genuine smiles and subsequent social outcomes.

Measuring trust with the Wayfinding Task: Implementing a novel task in immersive virtual reality and desktop setups across remote and in-person test environments

Trust is a key feature of social relationships. Common measures of trust, questionnaires and economic games, lack ecological validity. Hence, we sought to introduce an immersive, virtual reality (VR) measure for the behavioral assessment of trust across remote and in-person settings, building on the maze task of Hale et al. (2018). Our 'Wayfinding Task' consists of an interconnected urban environment for participants to navigate on the advice of two characters of differing trustworthiness. We present four studies implementing the Wayfinding Task in remote and in-person testing environments and comparing performance across head-mounted display (HMD)-based VR and desktop setups. In each study, the trustworthiness of two virtual characters was manipulated, through either a fact sheet providing trustworthiness information, or a behavior-based trustworthiness manipulation task termed the Door Game, based on Van der Biest et al., 2020. Participants then completed the Wayfinding Task. Overall, we found that participant behavior in the Wayfinding Task reflected the relative trustworthiness of the two characters; in particular, the trustworthy character was approached more often for advice, reflecting data from our Door Game. We found mostly null results for our novel outcome measure, interpersonal distance. Remote testing successfully achieved these effects. While HMD-based VR and desktop setups both showed these effects, there was a stronger effect of trustworthiness in the HMD VR version of the task. These results have implications for the measurement of trust in behavioral settings and the use of remote and VR-based testing in social experiments.

Virtual social interactions during the COVID-19 pandemic: the effect of interpersonal motor synchrony on social interactions in the virtual space

Although the link between motor synchrony and emotional alignment has been extensively studied during face-to-face interaction, the question of whether this association also exists in virtual settings has remained unanswered. Here, we examined whether this link exists during virtual social interactions and whether pro-social effects will be induced during those interactions. To this end, two strangers shared difficulties they have experienced due to the COVID-19 pandemic during a virtual social interaction that included both audio and video. The findings revealed that motor synchrony and emotional alignment can arise spontaneously during a virtual social interaction between two strangers. Moreover, this interaction yielded a decrease in negative affect and an increase in positive affect, as well as an increase in feelings of trust, liking, cohesion, self-other overlap, and similarity between the strangers. Finally, a higher level of synchrony during the virtual interaction was specifically associated with increased positive emotional alignment and liking. It can thus be presumed that virtual social interactions may share similar characteristics and social effects as face-to-face interactions. Considering the tremendous changes the COVID-19 pandemic has caused regarding social communication, these findings may provide grounds for developing new intervention protocols aimed at dealing with the consequences of social distancing.

How video calls affect mimicry and trust during interactions

Many social species, humans included, mimic emotional expressions, with important consequences for social bonding. Although humans increasingly interact via video calls, little is known about the effect of these online interactions on the mimicry of scratching and yawning, and their linkage with trust. The current study investigated whether mimicry and trust are affected by these new communication media. Using participant-confederate dyads (n = 27), we tested the mimicry of four behaviours across three different conditions: watching a pre-recorded video, online video call, and face-to-face. We measured mimicry of target behaviours frequently observed in emotional situations, yawn and scratch and control behaviours, lip-bite and face-touch. In addition, trust in the confederate was assessed via a trust game. Our study revealed that (i) mimicry and trust did not differ between face-to-face and video calls, but were significantly lower in the pre-recorded condition; and (ii) target behaviours were significantly more mimicked than the control behaviours. This negative relationship can possibly be explained by the negative connotation usually associated with the behaviours included in this study. Overall, this study showed that video calls might provide enough interaction cues for mimicry to occur in our student population and during interactions between strangers. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Neural mechanisms for emotional contagion and spontaneous mimicry of live facial expressions

Viewing a live facial expression typically elicits a similar expression by the observer (facial mimicry) that is associated with a concordant emotional experience (emotional contagion). The model of embodied emotion proposes that emotional contagion and facial mimicry are functionally linked although the neural underpinnings are not known. To address this knowledge gap, we employed a live two-person paradigm (n = 20 dyads) using functional near-infrared spectroscopy during live emotive face-processing while also measuring eye-tracking, facial classifications and ratings of emotion. One dyadic partner, 'Movie Watcher', was instructed to emote natural facial expressions while viewing evocative short movie clips. The other dyadic partner, 'Face Watcher', viewed the Movie Watcher's face. Task and rest blocks were implemented by timed epochs of clear and opaque glass that separated partners. Dyadic roles were alternated during the experiment. Mean cross-partner correlations of facial expressions (r = 0.36 ± 0.11 s.e.m.) and mean cross-partner affect ratings (r = 0.67 ± 0.04) were consistent with facial mimicry and emotional contagion, respectively. Neural correlates of emotional contagion based on covariates of partner affect ratings included angular and supramarginal gyri, whereas neural correlates of the live facial action units included motor cortex and ventral face-processing areas. Findings suggest distinct neural components for facial mimicry and emotional contagion. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

The Promises and Pitfalls of Virtual Reality

Increasingly, Virtual Reality technologies are finding a place in psychology and behavioral neuroscience labs. Immersing participants in virtual worlds enables researchers to investigate empirical questions in realistic or imaginary environments while measuring a wide range of behavioral responses, without sacrificing experimental control. In this chapter, we aim to provide a balanced appraisal of VR research methods. We describe how VR can help advance psychological science by opening pathways for addressing many pernicious challenges currently facing science (e.g., direct replication, prioritizing ecological validity). We also outline a range of unique and perhaps unanticipated obstacles and provide practical recommendations to overcome them.

Interviewing in virtual environments: Towards understanding the impact of rapport-building behaviours and retrieval context on eyewitness memory

Given the complexities of episodic memory and necessarily social nature of in-person face-to-face interviews, theoretical and evidence-based techniques for collecting episodic information from witnesses, victims, and survivors champion rapport-building. Rapport is believed to reduce some of the social demands of recalling an experienced event in an interview context, potentially increasing cognitive capacity for remembering. Cognitive and social benefits have also emerged in remote interview contexts with reduced anxiety and social pressure contributing to improved performance. Here, we investigated episodic memory in mock-eyewitness interviews conducted in virtual environments (VE) and in-person face-to-face (FtF), where rapport-building behaviours were either present or absent. Main effects revealed when rapport was present and where interviews were conducted in a VE participants recalled more correct event information, made fewer errors and were more accurate. Moreover, participants in the VE plus rapport-building present condition outperformed participants in all other conditions. Feedback indicated both rapport and environment were important for reducing the social demands of a recall interview, towards supporting effortful remembering. Our results add to the emerging literature on the utility of virtual environments as interview spaces and lend further support to the importance of prosocial behaviours in applied contexts.

Perspectives on Socially Intelligent Conversational Agents

The propagation of digital assistants is consistently progressing. Manifested by an uptake of ever more human-like conversational abilities, respective technologies are moving increasingly away from their role as voice-operated task enablers and becoming rather companion-like artifacts whose interaction style is rooted in anthropomorphic behavior. One of the required characteristics in this shift from a utilitarian tool to an emotional character is the adoption of social intelligence. Although past research has recognized this need, more multi-disciplinary investigations should be devoted to the exploration of relevant traits and their potential embedding in future agent technology. Aiming to lay a foundation for further developments, we report on the results of a Delphi study highlighting the respective opinions of 21 multi-disciplinary domain experts. Results exhibit 14 distinctive characteristics of social intelligence, grouped into different levels of consensus, maturity, and abstraction, which may be considered a relevant basis, assisting the definition and consequent development of socially intelligent conversational agents.

Social signalling as a framework for second-person neuroscience

Despite the recent increase in second-person neuroscience research, it is still hard to understand which neurocognitive mechanisms underlie real-time social behaviours. Here, we propose that social signalling can help us understand social interactions both at the single- and two-brain level in terms of social signal exchanges between senders and receivers. First, we show how subtle manipulations of being watched provide an important tool to dissect meaningful social signals. We then focus on how social signalling can help us build testable hypotheses for second-person neuroscience with the example of imitation and gaze behaviour. Finally, we suggest that linking neural activity to specific social signals will be key to fully understand the neurocognitive systems engaged during face-to-face interactions.

Collective Rhythm as an Emergent Property During Human Social Coordination

The literature on social interactions has shown that participants coordinate not only at the behavioral but also at the physiological and neural levels, and that this coordination gives a temporal structure to the individual and social dynamics. However, it has not been fully explored whether such temporal patterns emerge during interpersonal coordination beyond dyads, whether this phenomenon arises from complex cognitive mechanisms or from relatively simple rules of behavior, or which are the sociocultural processes that underlie this phenomenon. We review the evidence for the existence of group-level rhythmic patterns that result from social interactions and argue that the complexity of group dynamics can lead to temporal regularities that cannot be predicted from the individual periodicities: an emergent collective rhythm. Moreover, we use this interpretation of the literature to discuss how taking into account the sociocultural niche in which individuals develop can help explain the seemingly divergent results that have been reported on the social influences and consequences of interpersonal coordination. We make recommendations on further research to test these arguments and their relationship to the feeling of belonging and assimilation experienced during group dynamics.

A liar and a copycat: nonverbal coordination increases with lie difficulty

Studies of the nonverbal correlates of deception tend to examine liars' behaviours as independent from the behaviour of the interviewer, ignoring joint action. To address this gap, experiment 1 examined the effect of telling a truth and easy, difficult and very difficult lies on nonverbal coordination. Nonverbal coordination was measured automatically by applying a dynamic time warping algorithm to motion-capture data. In experiment 2, interviewees also received instructions that influenced the attention they paid to either the nonverbal or verbal behaviour of the interviewer. Results from both experiments found that interviewer-interviewee nonverbal coordination increased with lie difficulty. This increase was not influenced by the degree to which interviewees paid attention to their nonverbal behaviour, nor by the degree of interviewer's suspicion. Our findings are consistent with the broader proposition that people rely on automated processes such as mimicry when under cognitive load.

Unobtrusive tracking of interpersonal orienting and distance predicts the subjective quality of social interactions

Interpersonal coordination of behaviour is essential for smooth social interactions. Measures of interpersonal behaviour, however, often rely on subjective evaluations, invasive measurement techniques or gross measures of motion. Here, we constructed an unobtrusive motion tracking system that enables detailed analysis of behaviour at the individual and interpersonal levels, which we validated using wearable sensors. We evaluate dyadic measures of joint orienting and distancing, synchrony and gaze behaviours to summarize data collected during natural conversation and joint action tasks. Our results demonstrate that patterns of proxemic behaviours, rather than more widely used measures of interpersonal synchrony, best predicted the subjective quality of the interactions. Increased distance between participants predicted lower enjoyment, while increased joint orienting towards each other during cooperation correlated with increased effort reported by the participants. Importantly, the interpersonal distance was most informative of the quality of interaction when task demands and experimental control were minimal. These results suggest that interpersonal measures of behaviour gathered during minimally constrained social interactions are particularly sensitive for the subjective quality of social interactions and may be useful for interaction-based phenotyping for further studies.

Human-like communication in conversational agents: a literature review and research agenda

Purpose

Conversational agents (chatbots, avatars and robots) are increasingly substituting human employees in service encounters. Their presence offers many potential benefits, but customers are reluctant to engage with them. A possible explanation is that conversational agents do not make optimal use of communicative behaviors that enhance relational outcomes. The purpose of this paper is to identify which human-like communicative behaviors used by conversational agents have positive effects on relational outcomes and which additional behaviors could be investigated in future research.

Design/methodology/approach

This paper presents a systematic review of 61 articles that investigated the effects of communicative behaviors used by conversational agents on relational outcomes. A taxonomy is created of all behaviors investigated in these studies, and a research agenda is constructed on the basis of an analysis of their effects and a comparison with the literature on human-to-human service encounters.

Findings

The communicative behaviors can be classified along two dimensions: modality (verbal, nonverbal, appearance) and footing (similarity, responsiveness). Regarding the research agenda, it is noteworthy that some categories of behaviors show mixed results and some behaviors that are effective in human-to-human interactions have not yet been investigated in conversational agents.

Practical implications

By identifying potentially effective communicative behaviors in conversational agents, this study assists managers in optimizing encounters between conversational agents and customers.

Originality/value

This is the first study that develops a taxonomy of communicative behaviors in conversational agents and uses it to identify avenues for future research.

Achieving affective human-virtual agent communication by enabling virtual agents to imitate positive expressions

Affective communication, communicating with emotion, during face-to-face communication is critical for social interaction. Advances in artificial intelligence have made it essential to develop affective human–virtual agent communication. A person’s belief during human–virtual agent interaction that the agent is a computer program affects social-cognitive processes. Whether this belief interferes with affective communication is an open question. We hypothesized that the imitation of a positive emotional expression by a virtual agent induces a positive emotion, regardless of the belief. To test this hypothesis, we conducted an fMRI study with 39 healthy volunteers, who were made to believe that a virtual agent was either a person or a computer. They were instructed to smile, and immediately afterwards, the virtual agent displayed a positive, negative, or neutral expression. The participants reported a positive emotion only when their smile was imitated by the agent’s positive expression regardless of their belief. This imitation activated the participants’ medial prefrontal cortex and precuneus, which are involved in anthropomorphism and contingency, respectively. These results suggest that a positive congruent response by a virtual agent can overcome the effect of believing that the agent is a computer program and thus contribute to achieving affective human–virtual agent communication.

Brain activity during reciprocal social interaction investigated using conversational robots as control condition

We present a novel functional magnetic resonance imaging paradigm for second-person neuroscience. The paradigm compares a human social interaction (human-human interaction, HHI) to an interaction with a conversational robot (human-robot interaction, HRI). The social interaction consists of 1 min blocks of live bidirectional discussion between the scanned participant and the human or robot agent. A final sample of 21 participants is included in the corpus comprising physiological (blood oxygen level-dependent, respiration and peripheral blood flow) and behavioural (recorded speech from all interlocutors, eye tracking from the scanned participant, face recording of the human and robot agents) data. Here, we present the first analysis of this corpus, contrasting neural activity between HHI and HRI. We hypothesized that independently of differences in behaviour between interactions with the human and robot agent, neural markers of mentalizing (temporoparietal junction (TPJ) and medial prefrontal cortex) and social motivation (hypothalamus and amygdala) would only be active in HHI. Results confirmed significantly increased response associated with HHI in the TPJ, hypothalamus and amygdala, but not in the medial prefrontal cortex. Future analysis of this corpus will include fine-grained characterization of verbal and non-verbal behaviours recorded during the interaction to investigate their neural correlates. This article is part of the theme issue 'From social brains to social robots: applying neurocognitive insights to human-robot interaction'.

Using mimicry of body movements by a virtual agent to increase synchronization behavior and rapport in individuals with schizophrenia

Synchronization of behavior such as gestures or postures is assumed to serve crucial functions in social interaction but has been poorly studied to date in schizophrenia. Using a virtual collaborative environment (VCS), we tested 1) whether synchronization of behavior, i.e., the spontaneous initiation of gestures that are congruent with those of an interaction partner, was impaired in individuals with schizophrenia compared with healthy participants; 2) whether mimicry of the patients' body movements by the virtual interaction partner was associated with increased behavioral synchronization and rapport. 19 patients and 19 matched controls interacted with a virtual agent who either mimicked their head and torso movements with a delay varying randomly between 0.5 s and 4 s or did not mimic, and rated feelings of rapport toward the virtual agent after each condition. Both groups exhibited a higher and similar synchronization behavior of the virtual agent forearm movements when they were in the Mimicry condition rather than in the No-mimicry condition. In addition, both groups felt more comfortable with a mimicking virtual agent rather than a virtual agent not mimicking them suggesting that mimicry is able to increase rapport in individuals with schizophrenia. Our results suggest that schizophrenia cannot be considered anymore as a disorder of imitation, particularly as regards behavioral synchronization processes in social interaction contexts.

Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape

As virtual reality (VR) technology and systems become more commercially available and accessible, more and more psychologists are starting to integrate VR as part of their methods. This approach offers major advantages in experimental control, reproducibility, and ecological validity, but also has limitations and hidden pitfalls which may distract the novice user. This study aimed to guide the psychologist into the novel world of VR, reviewing available instrumentation and mapping the landscape of possible systems. We use examples of state‐of‐the‐art research to describe challenges which research is now solving, including embodiment, uncanny valley, simulation sickness, presence, ethics, and experimental design. Finally, we propose that the biggest challenge for the field would be to build a fully interactive virtual human who can pass a VR Turing test – and that this could only be achieved if psychologists, VR technologists, and AI researchers work together.

Synchrony and social connection in immersive Virtual Reality

Synchronising movements in time with others can have significant positive effects on affiliative attitudes and behaviors. To explore the generalizability of synchrony effects, and to eliminate confounds of suggestion, competence and shared intention typical of standard laboratory and field experiments, we used an Immersive Virtual Reality (VR) environment. Participants, represented as virtual humans, took part in a joint movement activity with two other programmed virtual humans. The timings of the co-participant characters' movements were covertly manipulated to achieve synchrony or non-synchrony with the focal participant. Participants in the synchrony condition reported significantly greater social closeness to their virtual co-participants than those in the non-synchrony condition. Results indicate that synchrony in joint action causes positive social effects and that these effects are robust in a VR setting. The research can potentially inform the development of VR interventions for social and psychological wellbeing.

The virtual maze: A behavioural tool for measuring trust

Trusting another person may depend on our level of generalised trust in others, as well as perceptions of that specific person’s trustworthiness. However, many studies measuring trust outcomes have not discussed generalised versus specific trust. To measure specific trust in others, we developed a novel behavioural task. Participants navigate a virtual maze and make a series of decisions about how to proceed. Before each decision, they may ask for advice from two virtual characters they have briefly interviewed earlier. We manipulated the virtual characters’ trustworthiness during the interview phase and measured how often participants approached and followed advice from each character. We also measured trust through ratings and an investment game. Across three studies, we found participants followed advice from a trustworthy character significantly more than an untrustworthy character, demonstrating the validity of the maze task. Behaviour in the virtual maze reflected specific trust rather than generalised trust, whereas the investment game picked up on generalised trust as well as specific trust. Our data suggest the virtual maze task may provide an alternative behavioural approach to measuring specific trust in future research, and we demonstrate how the task may be used in traditional laboratories.