Observation and imitation of actions performed by humans, androids, and robots: an EMG study

Does Action Observation of the Whole Task Influence Mirror Neuron System and Upper Limb Muscle Activity Better Than Part Task in People With Stroke?

Background: Task-based action observation and imitation (AOI) is a promising intervention to enhance upper limb (UL) motor function poststroke. However, whether whole/part task must be trained in the AOI therapy needs further substantiation. Objective: The objective of this study is to assess and compare the mirror neuron activity and UL muscle activity during AOI of reaching task in terms of whole task (complete movement) and part task (proximal arm movements and distal arm movements). Methods: In this cross-sectional study, 26 participants with first-time unilateral stroke were asked to observe the prerecorded videos of a reaching task in terms of a whole task and proximal and distal components, followed by imitation of the task, respectively. Electroencephalographic (EEG) mu rhythm suppression and electromyographic amplitude of six UL muscles were measured during the task. Results: The analysis of EEG revealed a statistically significant mu suppression score, indicating mirror neuron system activity, during AOI of the whole task in C3 (p = <0.001) and C4 (p = <0.001) electrodes compared to the part task. Percentage maximum voluntary contraction amplitudes of the deltoid (p = 0.002), supraspinatus (p = <0.001), triceps brachii (p = 0.002), brachioradialis (p = 0.006), and extensor carpi radialis (p = <0.001) muscles showed a significant increase in muscle activity during AOI of the whole task. Also, there seems to be a task observation-specific activation of muscles following AOI of proximal or distal tasks. Conclusion: The practice of the whole task should be given emphasis while framing the AOI treatment module to enhance reaching in people with stroke. Trial registration: Clinical Trials Registry-India (CTRI) identifier: CTRI/2018/04/013466.

Psychophysiological responses to eye contact with a humanoid robot: Impact of perceived intentionality

Eye contact with a social robot has been shown to elicit similar psychophysiological responses to eye contact with another human. However, it is becoming increasingly clear that the attention- and affect-related psychophysiological responses differentiate between direct (toward the observer) and averted gaze mainly when viewing embodied faces that are capable of social interaction, whereas pictorial or pre-recorded stimuli have no such capability. It has been suggested that genuine eye contact, as indicated by the differential psychophysiological responses to direct and averted gaze, requires a feeling of being watched by another mind. Therefore, we measured event-related potentials (N170 and frontal P300) with EEG, facial electromyography, skin conductance, and heart rate deceleration responses to seeing a humanoid robot's direct versus averted gaze, while manipulating the impression of the robot's intentionality. The results showed that the N170 and the facial zygomatic responses were greater to direct than to averted gaze of the robot, and independent of the robot's intentionality, whereas the frontal P300 responses were more positive to direct than to averted gaze only when the robot appeared intentional. The study provides further evidence that the gaze behavior of a social robot elicits attentional and affective responses and adds that the robot's seemingly autonomous social behavior plays an important role in eliciting higher-level socio-cognitive processing.

Stimuli in 3 Acts: A normative study on action-statements, action videos and object photos

The study of action observation and imagery, separately and combined, is expanding in diverse research areas (e.g., sports psychology, neurosciences), making clear the need for action-related stimuli (i.e., action statements, videos, and pictures). Although several databases of object and action pictures are available, norms on action videos are scarce. In this study, we validated a set of 60 object-related everyday actions in three different formats: action-statements, and corresponding dynamic (action videos) and static (object photos) stimuli. In Study 1, ratings of imageability, image agreement, action familiarity, action frequency, and action valence were collected from 161 participants. In Study 2, a different sample of 115 participants rated object familiarity, object valence, and object-action prototypicality. Most actions were rated as easy to imagine, familiar, and neutral or positive in valence. However, there was variation in the frequency with which participants perform these actions on a daily basis. High agreement between participants' mental image and action videos was also found, showing that the videos depict a conventional way of performing the actions. Objects were considered familiar and positive in valence. High ratings on object-action prototypicality indicate that the actions correspond to prototypical actions for most objects. 3ActStimuli is a comprehensive set of stimuli that can be useful in several research areas, allowing the combined study of action observation and imagery.

Human but not robotic gaze facilitates action prediction

Do people ascribe intentions to humanoid robots as they would to humans or non-human-like animated objects? In six experiments, we compared people’s ability to extract non-mentalistic (i.e., where an agent is looking) and mentalistic (i.e., what an agent is looking at; what an agent is going to do) information from gaze and directional cues performed by humans, human-like robots, and a non-human-like object. People were faster to infer the mental content of human agents compared to robotic agents. Furthermore, although the absence of differences in control conditions rules out the use of non-mentalizing strategies, the human-like appearance of non-human agents may engage mentalizing processes to solve the task. Overall, results suggest that human-like robotic actions may be processed differently from humans’ and objects’ behavior. These findings inform our understanding of the relevance of an object’s physical features in triggering mentalizing abilities and its relevance for human–robot interaction.

•

People differently ascribe mental content to human-like and non-human-like agents

•

A human-like shape may automatically engage mentalizing processes

•

Human actions are interpreted faster than non-human actions

Addressing joint action challenges in HRI: Insights from psychology and philosophy

The vast expansion of research in human-robot interactions (HRI) these last decades has been accompanied by the design of increasingly skilled robots for engaging in joint actions with humans. However, these advances have encountered significant challenges to ensure fluent interactions and sustain human motivation through the different steps of joint action. After exploring current literature on joint action in HRI, leading to a more precise definition of these challenges, the present article proposes some perspectives borrowed from psychology and philosophy showing the key role of communication in human interactions. From mutual recognition between individuals to the expression of commitment and social expectations, we argue that communicative cues can facilitate coordination, prediction, and motivation in the context of joint action. The description of several notions thus suggests that some communicative capacities can be implemented in the context of joint action for HRI, leading to an integrated perspective of robotic communication.

Action observation and imitation in Parkinson's disease: The influence of biological and non-biological stimuli

Action observation and imitation have been found to influence movement in people with Parkinson's disease (PD), but simple visual stimuli can also guide their movement. To investigate whether action observation may provide a more effective stimulus than other visual cues, the present study examined the effects of observing human pointing movements and simple visual stimuli on hand kinematics and eye movements in people with mild to moderate PD and age-matched controls. In Experiment 1, participants observed videos of movement sequences between horizontal positions, depicted by a simple cue with or without a moving human hand, then imitated the sequence either without further visual input (consecutive task) or while watching the video again (concurrent task). Modulation of movement duration, in accordance with changes in the observed stimulus, increased when the simple cue was accompanied by the hand and in the concurrent task, whereas modulation of horizontal amplitude was greater with the simple cue alone and in the consecutive task. Experiment 2 compared imitation of kinematically-matched dynamic biological (human hand) and non-biological (shape) stimuli, which moved with a high or low vertical trajectory. Both groups exhibited greater modulation for the hand than the shape, and differences in eye movements suggested closer tracking of the hand. Despite producing slower and smaller movements overall, the PD group showed a similar pattern of imitation to controls across tasks and conditions. The findings demonstrate that observing human action influences aspects of movement such as duration or trajectory more strongly than non-biological stimuli, particularly during concurrent imitation.

Making eye contact with a robot: Psychophysiological responses to eye contact with a human and with a humanoid robot

Previous research has shown that eye contact, in human-human interaction, elicits increased affective and attention related psychophysiological responses. In the present study, we investigated whether eye contact with a humanoid robot would elicit these responses. Participants were facing a humanoid robot (NAO) or a human partner, both physically present and looking at or away from the participant. The results showed that both in human-robot and human-human condition, eye contact versus averted gaze elicited greater skin conductance responses indexing autonomic arousal, greater facial zygomatic muscle responses (and smaller corrugator responses) associated with positive affect, and greater heart deceleration responses indexing attention allocation. With regard to the skin conductance and zygomatic responses, the human model's gaze direction had a greater effect on the responses as compared to the robot's gaze direction. In conclusion, eye contact elicits automatic affective and attentional reactions both when shared with a humanoid robot and with another human.

The Mimicry Among Us: Intra- and Inter-Personal Mechanisms of Spontaneous Mimicry

This review explores spontaneous mimicry in the context of three questions. The first question concerns the role of spontaneous mimicry in processing conceptual information. The second question concerns the debate whether spontaneous mimicry is driven by simple associative processes or reflects higher-order processes such as goals, intentions, and social context. The third question addresses the implications of these debates for understanding atypical individuals and states. We review relevant literature and argue for a dynamic, context-sensitive role of spontaneous mimicry in social cognition and behavior. We highlight how the modulation of mimicry is often adaptive but also point out some cases of maladaptive modulations that impair an individuals’ engagement in social life.

Pupillary Responses to Robotic and Human Emotions: The Uncanny Valley and Media Equation Confirmed

Physiological responses during human–robots interaction are useful alternatives to subjective measures of uncanny feelings for nearly humanlike robots (uncanny valley) and comparable emotional responses between humans and robots (media equation). However, no studies have employed the easily accessible measure of pupillometry to confirm the uncanny valley and media equation hypotheses, evidence in favor of the existence of these hypotheses in interaction with emotional robots is scarce, and previous studies have not controlled for low level image statistics across robot appearances. We therefore recorded pupil size of 40 participants that viewed and rated pictures of robotic and human faces that expressed a variety of basic emotions. The robotic faces varied along the dimension of human likeness from cartoonish to humanlike. We strictly controlled for confounding factors by removing backgrounds, hair, and color, and by equalizing low level image statistics. After the presentation phase, participants indicated to what extent the robots appeared uncanny and humanlike, and whether they could imagine social interaction with the robots in real life situations. The results show that robots rated as nearly humanlike scored higher on uncanniness, scored lower on imagined social interaction, evoked weaker pupil dilations, and their emotional expressions were more difficult to recognize. Pupils dilated most strongly to negative expressions and the pattern of pupil responses across emotions was highly similar between robot and human stimuli. These results highlight the usefulness of pupillometry in emotion studies and robot design by confirming the uncanny valley and media equation hypotheses.

From automata to animate beings: the scope and limits of attributing socialness to artificial agents

Understanding the mechanisms and consequences of attributing socialness to artificial agents has important implications for how we can use technology to lead more productive and fulfilling lives. Here, we integrate recent findings on the factors that shape behavioral and brain mechanisms that support social interactions between humans and artificial agents. We review how visual features of an agent, as well as knowledge factors within the human observer, shape attributions across dimensions of socialness. We explore how anthropomorphism and dehumanization further influence how we perceive and interact with artificial agents. Based on these findings, we argue that the cognitive reconstruction within the human observer is likely to be far more crucial in shaping our interactions with artificial agents than previously thought, while the artificial agent's visual features are possibly of lesser importance. We combine these findings to provide an integrative theoretical account based on the "like me" hypothesis, and discuss the key role played by the Theory-of-Mind network, especially the temporal parietal junction, in the shift from mechanistic to social attributions. We conclude by highlighting outstanding questions on the impact of long-term interactions with artificial agents on the behavioral and brain mechanisms of attributing socialness to these agents.

Behind the Robot's Smiles and Frowns: In Social Context, People Do Not Mirror Android's Expressions But React to Their Informational Value

Facial actions are key elements of non-verbal behavior. Perceivers' reactions to others' facial expressions often represent a match or mirroring (e.g., they smile to a smile). However, the information conveyed by an expression depends on context. Thus, when shown by an opponent, a smile conveys bad news and evokes frowning. The availability of anthropomorphic agents capable of facial actions raises the question of how people respond to such agents in social context. We explored this issue in a study where participants played a strategic game with or against a facially expressive android. Electromyography (EMG) recorded participants' reactions over zygomaticus muscle (smiling) and corrugator muscle (frowning). We found that participants' facial responses to android's expressions reflect their informational value, rather than a direct match. Overall, participants smiled more, and frowned less, when winning than losing. Critically, participants' responses to the game outcome were similar regardless of whether it was conveyed via the android's smile or frown. Furthermore, the outcome had greater impact on people's facial reactions when it was conveyed through android's face than a computer screen. These findings demonstrate that facial actions of artificial agents impact human facial responding. They also suggest a sophistication in human-robot communication that highlights the signaling value of facial expressions.

Mimicking and anticipating others' actions is linked to Social Information Processing

It is widely known that individuals frequently imitate each other in social situations and that such mimicry fulfills an important social role in the sense that it functions as a social glue. With reference to the anticipated action effect, it has recently been demonstrated that individuals do not only imitate others, but also engage in anticipated action before the observed person starts engaging in that action. Interestingly, both phenomena (i.e., mimicry and anticipated action) rely on tracking others’ social behavior. Therefore, in the present research we investigated whether mimicry and anticipated action are related to social abilities as indicated by measures of social intelligence. The results demonstrate for the first time that mimicry as well as anticipated action is correlated with an important aspect of social intelligence—namely the ability to process social information. Theoretical implications and limitations are discussed.

Fronto-parietal coding of goal-directed actions performed by artificial agents

With advances in technology, artificial agents such as humanoid robots will soon become a part of our daily lives. For safe and intuitive collaboration, it is important to understand the goals behind their motor actions. In humans, this process is mediated by changes in activity in fronto-parietal brain areas. The extent to which these areas are activated when observing artificial agents indicates the naturalness and easiness of interaction. Previous studies indicated that fronto-parietal activity does not depend on whether the agent is human or artificial. However, it is unknown whether this activity is modulated by observing grasping (self-related action) and pointing actions (other-related action) performed by an artificial agent depending on the action goal. Therefore, we designed an experiment in which subjects observed human and artificial agents perform pointing and grasping actions aimed at two different object categories suggesting different goals. We found a signal increase in the bilateral inferior parietal lobule and the premotor cortex when tool versus food items were pointed to or grasped by both agents, probably reflecting the association of hand actions with the functional use of tools. Our results show that goal attribution engages the fronto-parietal network not only for observing a human but also a robotic agent for both self-related and social actions. The debriefing after the experiment has shown that actions of human-like artificial agents can be perceived as being goal-directed. Therefore, humans will be able to interact with service robots intuitively in various domains such as education, healthcare, public service, and entertainment.

Mimicry and automatic imitation are not correlated

It is widely known that individuals have a tendency to imitate each other. However, different psychological disciplines assess imitation in different manners. While social psychologists assess mimicry by means of action observation, cognitive psychologists assess automatic imitation with reaction time based measures on a trial-by-trial basis. Although these methods differ in crucial methodological aspects, both phenomena are assumed to rely on similar underlying mechanisms. This raises the fundamental question whether mimicry and automatic imitation are actually correlated. In the present research we assessed both phenomena and did not find a meaningful correlation. Moreover, personality traits such as empathy, autism traits, and traits related to self- versus other-focus did not correlate with mimicry or automatic imitation either. Theoretical implications are discussed.

The Role of the Human Mirror Neuron System in Supporting Communication in a Digital World

Humans use both verbal and non-verbal communication to interact with others and their environment and increasingly these interactions are occurring in a digital medium. Whether live or digital, learning to communicate requires overcoming the correspondence problem: There is no direct mapping, or correspondence between perceived and self-produced signals. Reconciliation of the differences between perceived and produced actions, including linguistic actions, is difficult and requires integration across multiple modalities and neuro-cognitive networks. Recent work on the neural substrates of social learning suggests that there may be a common mechanism underlying the perception-production cycle for verbal and non-verbal communication. The purpose of this paper is to review evidence supporting the link between verbal and non-verbal communications, and to extend the hMNS literature by proposing that recent advances in communication technology, which at times have had deleterious effects on behavioral and perceptual performance, may disrupt the success of the hMNS in supporting social interactions because these technologies are virtual and spatiotemporal distributed nature.

Thinking Outside the Button Box: EMG as a Computer Input Device for Psychological Research

Experimental psychology research commonly has participants respond to stimuli by pressing buttons or keys. Standard computer input devices constrain the range of motoric responses participants can make, even as the field advances theory about the importance of the motor system in cognitive and social information processing. Here we describe an inexpensive way to use an electromyographic (EMG) signal as a computer input device, enabling participants to control a computer by contracting muscles that are not usually used for that purpose, but which may be theoretically relevant. We tested this approach in a study of facial mimicry, a well-documented phenomenon in which viewing emotional faces elicits automatic activation of corresponding muscles in the face of the viewer. Participants viewed happy and angry faces and were instructed to indicate the emotion on each face as quickly as possible by either furrowing their brow or contracting their cheek. The mapping of motor response to judgment was counterbalanced, so that one block of trials required a congruent mapping (contract brow to respond “angry,” cheek to respond “happy”) and the other block required an incongruent mapping (brow for “happy,” cheek for “angry”). EMG sensors placed over the left corrugator supercilii muscle and left zygomaticus major muscle fed readings of muscle activation to a microcontroller, which sent a response to a computer when activation reached a pre-determined threshold. Response times were faster when the motor-response mapping was congruent than when it was incongruent, extending prior studies on facial mimicry. We discuss further applications of the method for research that seeks to expand the range of human-computer interaction beyond the button box.