Cooperation or competition? Discriminating between social intentions by observing prehensile movements

Processing of visual social-communication cues during a social-perception of action task in autistic and non-autistic observers

Social perception and communication differ between those with and without autism, even when verbal fluency and intellectual ability are equated. Previous work found that observers responded more quickly to an actor's points if the actor had chosen by themselves where to point instead of being directed where to point. Notably, this 'choice-advantage' effect decreased across non-autistic participants as the number of autistic-like traits and tendencies increased (Pesquita et al., 2016). Here, we build on that work using the same task to study individuals over a broader range of the spectrum, from autistic to non-autistic, measuring both response initiation and mouse movement times, and considering the response to each actor separately. Autistic and non-autistic observers viewed videos of three different actors pointing to one of two locations, without knowing that the actors were sometimes freely choosing to point to one target and other times being directed where to point. All observers exhibited a choice-advantage overall, meaning they responded more rapidly when actors were freely choosing versus when they were directed, indicating a sensitivity to the actors' postural cues and movements. Our fine-grained analyses found a more robust choice-advantage to some actors than others, with autistic observers showing a choice-advantage only in response to one of the actors, suggesting that both actor and observer characteristics influence the overall effect. We briefly explore existing actor characteristics that may have contributed to this effect, finding that both duration of exposure to pre-movement cues and kinematic cues of the actors likely influence the choice advantage to different degrees across the groups. Altogether, the evidence suggested that both autistic and non-autistic individuals could detect the choice-advantage signal, but that for autistic observers the choice-advantage was actor specific. Notably, we found that the influence of the signal, when present, was detected early for all actors by the non-autistic observers, but detected later and only for one actor by the autistic observers. Altogether, we have more accurately characterized the ability of social-perception in autistic individuals as intact, but highlighted that detection of signal is likely delayed/distributed compared to non-autistic observers and that it is important to investigate actor characteristics that may influence detection and use of their social-perception signals.

Exploring the use of visual predictions in social scenarios while under anticipatory threat

One of the less recognized effects of anxiety lies in perception alterations caused by how one weighs both sensory evidence and contextual cues. Here, we investigated how anxiety affects our ability to use social cues to anticipate the others' actions. We adapted a paradigm to assess expectations in social scenarios, whereby participants were asked to identify the presence of agents therein, while supported by contextual cues from another agent. Participants (N = 66) underwent this task under safe and threat-of-shock conditions. We extracted both criterion and sensitivity measures as well as gaze data. Our analysis showed that whilst the type of action had the expected effect, threat-of-shock had no effect over criterion and sensitivity. Although showing similar dwell times, gaze exploration of the contextual cue was associated with shorter fixation durations whilst participants were under threat. Our findings suggest that anxiety does not appear to influence the use of expectations in social scenarios.

Contextual Priors Shape Action Understanding before and beyond the Unfolding of Movement Kinematics

Previous studies have shown that contextual information may aid in guessing the intention underlying others' actions in conditions of perceptual ambiguity. Here, we aimed to evaluate the temporal deployment of contextual influence on action prediction with increasing availability of kinematic information during the observation of ongoing actions. We used action videos depicting an actor grasping an object placed on a container to perform individual or interpersonal actions featuring different kinematic profiles. Crucially, the container could be of different colors. First, in a familiarization phase, the probability of co-occurrence between each action kinematics and color cues was implicitly manipulated to 80% and 20%, thus generating contextual priors. Then, in a testing phase, participants were asked to predict action outcome when the same action videos were occluded at five different timeframes of the entire movement, ranging from when the actor was still to when the grasp of the object was fully accomplished. In this phase, all possible action-contextual cues' associations were equally presented. The results showed that for all occlusion intervals, action prediction was more facilitated when action kinematics deployed in high- than low-probability contextual scenarios. Importantly, contextual priors shaped action prediction even in the latest occlusion intervals, where the kinematic cues clearly unveiled an action outcome that was previously associated with low-probability scenarios. These residual contextual effects were stronger in individuals with higher subclinical autistic traits. Our findings highlight the relative contribution of kinematic and contextual information to action understanding and provide evidence in favor of their continuous integration during action observation.

Individual differences in cooperative and competitive play strategies

INTRODUCTION

Cooperation and competition are common in social interactions. It is not clear how individual differences in personality may predict performance strategies under these two contexts. We evaluated whether instructions to play cooperatively and competitively would differentially affect dyads playing a Pong video game. We hypothesized that instructions to play cooperatively would result in lower overall points scored and differences in paddle control kinematics relative to when participants were instructed to play competitively. We also predicted that higher scores in prosociality and Sportspersonship would be related to better performance during cooperative than competitive conditions.

METHODS

Pairs of participants played a Pong video game under cooperative and competitive instructions. During competitive trials, participants were instructed to score more points against one another to win the game. During the cooperative trials, participants were instructed to work together to score as few points against one another as possible. After game play, each participant completed surveys so we could measure their trait prosociality and Sportspersonship.

RESULTS

Condition was a significant predictor of where along the paddle participants hit the ball, which controlled ball exit angles. Specifically, during cooperation participants concentrated ball contacts on the paddle towards the center to produce more consistent rebound angles. We found a significant correlation of Sex and the average points scored by participants during cooperative games, competitive games, and across all trials. Sex was also significantly correlated with paddle kinematics during cooperative games. The overall scores on the prosociality and Sportspersonship surveys were not significantly correlated with the performance outcomes in cooperative and competitive games. The dimension of prosociality assessing empathic concern was significantly correlated with performance outcomes during cooperative video game play.

DISCUSSION

No Sportspersonship survey score was able to predict cooperative or competitive game performance, suggesting that Sportspersonship personality assessments are not reliable predictors of cooperative or competitive behaviors translated to a virtual game setting. Survey items and dimensions probing broader empathic concern may be more effective predictors of cooperative and competitive performance during interactive video game play. Further testing is encouraged to assess the efficacy of prosocial personality traits as predictors of cooperative and competitive video game behavior.

What's your point? Insights from virtual reality on the relation between intention and action in the production of pointing gestures

Human communication involves the process of translating intentions into communicative actions. But how exactly do our intentions surface in the visible communicative behavior we display? Here we focus on pointing gestures, a fundamental building block of everyday communication, and investigate whether and how different types of underlying intent modulate the kinematics of the pointing hand and the brain activity preceding the gestural movement. In a dynamic virtual reality environment, participants pointed at a referent to either share attention with their addressee, inform their addressee, or get their addressee to perform an action. Behaviorally, it was observed that these different underlying intentions modulated how long participants kept their arm and finger still, both prior to starting the movement and when keeping their pointing hand in apex position. In early planning stages, a neurophysiological distinction was observed between a gesture that is used to share attitudes and knowledge with another person versus a gesture that mainly uses that person as a means to perform an action. Together, these findings suggest that our intentions influence our actions from the earliest neurophysiological planning stages to the kinematic endpoint of the movement itself.

Depth and direction effects in the prediction of static and shifted reaching goals from kinematics

The kinematic parameters of reach-to-grasp movements are modulated by action intentions. However, when an unexpected change in visual target goal during reaching execution occurs, it is still unknown whether the action intention changes with target goal modification and which is the temporal structure of the target goal prediction. We recorded the kinematics of the pointing finger and wrist during the execution of reaching movements in 23 naïve volunteers where the targets could be located at different directions and depths with respect to the body. During the movement execution, the targets could remain static for the entire duration of movement or shifted, with different timings, to another position. We performed temporal decoding of the final goals and of the intermediate trajectory from the past kinematics exploiting a recurrent neural network. We observed a progressive increase of the classification performance from the onset to the end of movement in both horizontal and sagittal dimensions, as well as in decoding shifted targets. The classification accuracy in decoding horizontal targets was higher than the classification accuracy of sagittal targets. These results are useful for establishing how human and artificial agents could take advantage from the observed kinematics to optimize their cooperation in three-dimensional space.

Motor invariants in action execution and perception

The nervous system is sensitive to statistical regularities of the external world and forms internal models of these regularities to predict environmental dynamics. Given the inherently social nature of human behavior, being capable of building reliable predictive models of others' actions may be essential for successful interaction. While social prediction might seem to be a daunting task, the study of human motor control has accumulated ample evidence that our movements follow a series of kinematic invariants, which can be used by observers to reduce their uncertainty during social exchanges. Here, we provide an overview of the most salient regularities that shape biological motion, examine the role of these invariants in recognizing others' actions, and speculate that anchoring socially-relevant perceptual decisions to such kinematic invariants provides a key computational advantage for inferring conspecifics' goals and intentions.

Understanding others' distal goals from proximal communicative actions

Many social interactions require individuals to coordinate their actions and to inform each other about their goals. Often these goals concern an immediate (i.e., proximal) action, as when people give each other a brief handshake, but they sometimes also refer to a future (i.e. distal) action, as when football players perform a passing sequence. The present study investigates whether observers can derive information about such distal goals by relying on kinematic modulations of an actor's instrumental actions. In Experiment 1 participants were presented with animations of a box being moved at different velocities towards an apparent endpoint. The distal goal, however, was for the object to be moved past this endpoint, to one of two occluded target locations. Participants then selected the location which they considered the likely distal goal of the action. As predicted, participants were able to detect differences in movement velocity and, based on these differences, systematically mapped the movements to the two distal goal locations. Adding a distal goal led to more variation in the way participants mapped the observed movements onto different target locations. The results of Experiments 2 and 3 indicated that this cannot be explained by difficulties in perceptual discrimination. Rather, the increased variability likely reflects differences in interpreting the underlying connection between proximal communicative actions and distal goals. The present findings extend previous research on sensorimotor communication by demonstrating that communicative action modulations are not restricted to predicting proximal goals but can also be used to infer more distal goals.

Measuring the prediction of observed actions using an occlusion paradigm: Comparing autistic and non-autistic adults

Action prediction involves observing and predicting the actions of others and plays an important role in social cognition and interacting with others. It is thought to use simulation, whereby the observers use their own motor system to predict the observed actions. As individuals diagnosed with autism are characterized by difficulties understanding the actions of others and motor coordination issues, it is possible that action prediction ability is altered in this population. This study compared action prediction ability between 20 autistic and 22 non-autistic adults using an occlusion paradigm. Participants watched different videos of a female actor carrying out everyday actions. During each video, the action was transiently occluded by a gray rectangle for 1000 ms. During occlusions, the video was allowed to continue as normal or was moved forward (i.e., appearing to continue too far ahead) or moved backwards (i.e., appearing to continue too far behind). Participants were asked to indicate after each occlusion whether the action continued with the correct timing or was too far ahead/behind. Autistic individuals were less accurate than non-autistic individuals, particularly when the video was too far behind. A trend analysis suggested that autistic participants were more likely to judge too far behind occlusions as being in time. These preliminary results suggest that prediction ability may be altered in autistic adults, potentially due to slower simulation or a delayed onset of these processes. LAY SUMMARY: When we observe other people performing everyday actions, we use their movements to help us understand and predict what they are doing. In this study, we found that autistic compared to non-autistic adults were slightly less accurate at predicting other people's actions. These findings help to unpick the different ways that social understanding is affected in autism.

Kinematic-Based Classification of Social Gestures and Grasping by Humans and Machine Learning Techniques

The affective motion of humans conveys messages that other humans perceive and understand without conventional linguistic processing. This ability to classify human movement into meaningful gestures or segments plays also a critical role in creating social interaction between humans and robots. In the research presented here, grasping and social gesture recognition by humans and four machine learning techniques (k-Nearest Neighbor, Locality-Sensitive Hashing Forest, Random Forest and Support Vector Machine) is assessed by using human classification data as a reference for evaluating the classification performance of machine learning techniques for thirty hand/arm gestures. The gestures are rated according to the extent of grasping motion on one task and the extent to which the same gestures are perceived as social according to another task. The results indicate that humans clearly rate differently according to the two different tasks. The machine learning techniques provide a similar classification of the actions according to grasping kinematics and social quality. Furthermore, there is a strong association between gesture kinematics and judgments of grasping and the social quality of the hand/arm gestures. Our results support previous research on intention-from-movement understanding that demonstrates the reliance on kinematic information for perceiving the social aspects and intentions in different grasping actions as well as communicative point-light actions.

Costs and benefits of communicating vigor

Why do we run toward people we love, but only walk toward others? One reason is to let them know we love them. In this commentary, we elaborate on how subjective utility information encoded in vigor is read out by others. We consider the potential implications for understanding and modeling the link between movements and decisions in social environments.

Action Observation Responses Are Influenced by Movement Kinematics and Target Identity

In order to inform the debate whether cortical areas related to action observation provide a pragmatic or a semantic representation of goal-directed actions, we performed 2 functional magnetic resonance imaging (fMRI) experiments in humans. The first experiment, involving observation of aimless arm movements, resulted in activation of most of the components known to support action execution and action observation. Given the absence of a target/goal in this experiment and the activation of parieto-premotor cortical areas, which were associated in the past with direction, amplitude, and velocity of movement of biological effectors, our findings suggest that during action observation we could be monitoring movement kinematics. With the second, double dissociation fMRI experiment, we revealed the components of the observation-related cortical network affected by 1) actions that have the same target/goal but different reaching and grasping kinematics and 2) actions that have very similar kinematics but different targets/goals. We found that certain areas related to action observation, including the mirror neuron ones, are informed about movement kinematics and/or target identity, hence providing a pragmatic rather than a semantic representation of goal-directed actions. Overall, our findings support a process-driven simulation-like mechanism of action understanding, in agreement with the theory of motor cognition, and question motor theories of action concept processing.

Acting with shared intentions: A systematic review on joint action coordination in Autism Spectrum Disorder

BACKGROUND

Joint actions, described as a form of social interaction in which individuals coordinate their actions in space and time to bring about a change in the environment, rely on sensory-motor processes that play a role in the development of social skills. Two brain networks, associated with "mirroring" and "mentalizing", are engaged during these actions: the mirror neuron and the theory of mind systems. People with autism spectrum disorder (ASD) showed impairment in interpersonal coordination during joint actions. Studying joint action coordination in ASD will contribute to clarify the interplay between sensory-motor and social processes throughout development and the interactions between the brain and the behavior.

METHOD

This review focused on empirical studies that reported behavioral and kinematic findings related to joint action coordination in people with ASD.

RESULTS

Literature on mechanisms involved in the joint action coordination impairment in ASD is still limited. Data are controversial. Different key-components of joint action coordination may be impaired, such as cooperative behavior, temporal coordination, and motor planning.

CONCLUSIONS

Interpersonal coordination during joint actions relies on early sensory-motor processes that have a key role in guiding social development. Early intervention targeting the sensory-motor processes involved in the development of joint action coordination could positively support social skills.

The communicative advantage: how kinematic signaling supports semantic comprehension

Humans are unique in their ability to communicate information through representational gestures which visually simulate an action (eg. moving hands as if opening a jar). Previous research indicates that the intention to communicate modulates the kinematics (e.g., velocity, size) of such gestures. If and how this modulation influences addressees' comprehension of gestures have not been investigated. Here we ask whether communicative kinematic modulation enhances semantic comprehension (i.e., identification) of gestures. We additionally investigate whether any comprehension advantage is due to enhanced early identification or late identification. Participants (n = 20) watched videos of representational gestures produced in a more- (n = 60) or less-communicative (n = 60) context and performed a forced-choice recognition task. We tested the isolated role of kinematics by removing visibility of actor's faces in Experiment I, and by reducing the stimuli to stick-light figures in Experiment II. Three video lengths were used to disentangle early identification from late identification. Accuracy and response time quantified main effects. Kinematic modulation was tested for correlations with task performance. We found higher gesture identification performance in more- compared to less-communicative gestures. However, early identification was only enhanced within a full visual context, while late identification occurred even when viewing isolated kinematics. Additionally, temporally segmented acts with more post-stroke holds were associated with higher accuracy. Our results demonstrate that communicative signaling, interacting with other visual cues, generally supports gesture identification, while kinematic modulation specifically enhances late identification in the absence of other cues. Results provide insights into mutual understanding processes as well as creating artificial communicative agents.

Predictable real-time constraints reveal anticipatory strategies of coupled planning in a sequential pick and place task

Planning a sequence of two motor elements is much more than concatenating two independent movements. However, very little is known about the cognitive strategies that are used to perform fluent sequences for intentional object manipulation. In this series of studies, the participants’ task was to reach for and pick to place a wooden cylinder to set it on a place pad of three different diameters, which served to modify terminal accuracy constraints. Participants were required to perform the sequences (1) at their preferred speed or (2) as fast as possible. Action kinematics were recorded with the Qualisys motion-capture system in order to implement a real-time protocol to get participants to engage in a true interactive relation. Results revealed that with low internal constraints (at preferred speed), low coupling between the two elements of the motor sequence was observed, suggesting a step-by-step planning strategy. Under high constraints (at fastest speed), an important terminal accuracy effect back propagated to modify early kinematic parameters of the first element, suggesting strong coupling of the parameters in an encapsulated planning strategy. In Studies 2 and 3, we further manipulated instructions and timing constraints to confirm the importance of time and predictability of external information for coupled planning. These findings overall sustain the hypothesis that coupled planning can take place in a pick and place task when anticipatory strategies are possible. This mode of action planning may be the key reason why motor intention can be read through the observation of micro variations in body kinematics.

Seeing the Unexpected: How Brains Read Communicative Intent through Kinematics

Social interaction requires us to recognize subtle cues in behavior, such as kinematic differences in actions and gestures produced with different social intentions. Neuroscientific studies indicate that the putative mirror neuron system (pMNS) in the premotor cortex and mentalizing system (MS) in the medial prefrontal cortex support inferences about contextually unusual actions. However, little is known regarding the brain dynamics of these systems when viewing communicatively exaggerated kinematics. In an event-related functional magnetic resonance imaging experiment, 28 participants viewed stick-light videos of pantomime gestures, recorded in a previous study, which contained varying degrees of communicative exaggeration. Participants made either social or nonsocial classifications of the videos. Using participant responses and pantomime kinematics, we modeled the probability of each video being classified as communicative. Interregion connectivity and activity were modulated by kinematic exaggeration, depending on the task. In the Social Task, communicativeness of the gesture increased activation of several pMNS and MS regions and modulated top-down coupling from the MS to the pMNS, but engagement of the pMNS and MS was not found in the nonsocial task. Our results suggest that expectation violations can be a key cue for inferring communicative intention, extending previous findings from wholly unexpected actions to more subtle social signaling.

Observer-Agent Kinematic Similarity Facilitates Action Intention Decoding

It is well known that the kinematics of an action is modulated by the underlying motor intention. In turn, kinematics serves as a cue also during action observation, providing hints about the intention of the observed action. However, an open question is whether decoding others’ intentions on the basis of their kinematics depends solely on how much the kinematics varies across different actions, or rather it is also influenced by its similarity with the observer motor repertoire. The execution of reach-to-grasp and place actions, differing for target size and context, was recorded in terms of upper-limb kinematics in 21 volunteers and in an actor. Volunteers had later to observe the sole reach-to-grasp phase of the actor’s actions, and predict the underlying intention. The potential benefit of the kinematic actor-participant similarity for recognition accuracy was evaluated. In execution, both target size and context modulated specific kinematic parameters. More importantly, although participants performed above chance in intention recognition, the similarity of motor patterns positively correlated with recognition accuracy. Overall, these data indicate that kinematic similarity exerts a facilitative role in intention recognition, providing further support to the view of action intention recognition as a visuo-motor process grounded in motor resonance.

Subtle predictive movements reveal actions regardless of social context

Humans have a remarkable ability to predict the actions of others. To address what information enables this prediction and how the information is modulated by social context, we used videos collected during an interactive reaching game. Two participants (an “initiator” and a “responder”) sat on either side of a plexiglass screen on which two targets were affixed. The initiator was directed to tap one of the two targets, and the responder had to either beat the initiator to the target (competition) or arrive at the same time (cooperation). In a psychophysics experiment, new observers predicted the direction of the initiators' reach from brief clips, which were clipped relative to when the initiator began reaching. A machine learning classifier performed the same task. Both humans and the classifier were able to determine the direction of movement before the finger lift-off in both social conditions. Further, using an information mapping technique, the relevant information was found to be distributed throughout the body of the initiator in both social conditions. Our results indicate that we reveal our intentions during cooperation, in which communicating the future course of actions is beneficial, and also during competition despite the social motivation to reveal less information.

What Can You See? Identifying Cues on Internal States From the Movements of Natural Social Interactions

In recent years, the field of Human-Robot Interaction (HRI) has seen an increasing demand for technologies that can recognize and adapt to human behaviors and internal states (e.g., emotions and intentions). Psychological research suggests that human movements are important for inferring internal states. There is, however, a need to better understand what kind of information can be extracted from movement data, particularly in unconstrained, natural interactions. The present study examines which internal states and social constructs humans identify from movement in naturalistic social interactions. Participants either viewed clips of the full scene or processed versions of it displaying 2D positional data. Then, they were asked to fill out questionnaires assessing their social perception of the viewed material. We analyzed whether the full scene clips were more informative than the 2D positional data clips. First, we calculated the inter-rater agreement between participants in both conditions. Then, we employed machine learning classifiers to predict the internal states of the individuals in the videos based on the ratings obtained. Although we found a higher inter-rater agreement for full scenes compared to positional data, the level of agreement in the latter case was still above chance, thus demonstrating that the internal states and social constructs under study were identifiable in both conditions. A factor analysis run on participants' responses showed that participants identified the constructs interaction imbalance, interaction valence and engagement regardless of video condition. The machine learning classifiers achieved a similar performance in both conditions, again supporting the idea that movement alone carries relevant information. Overall, our results suggest it is reasonable to expect a machine learning algorithm, and consequently a robot, to successfully decode and classify a range of internal states and social constructs using low-dimensional data (such as the movements and poses of observed individuals) as input.

Increased functional coupling of the left amygdala and medial prefrontal cortex during the perception of communicative point-light stimuli

Interpersonal predictive coding (IPPC) describes the behavioral phenomenon whereby seeing a communicative rather than an individual action helps to discern a masked second agent. As little is known, yet, about the neural correlates of IPPC, we conducted a functional magnetic resonance imaging study in a group of 27 healthy participants using point-light displays of moving agents embedded in distractors. We discovered that seeing communicative compared to individual actions was associated with higher activation of right superior frontal gyrus, whereas the reversed contrast elicited increased neural activation in an action observation network that was activated during all trials. Our findings, therefore, potentially indicate the formation of action predictions and a reduced demand for executive control in response to communicative actions. Further, in a regression analysis, we revealed that increased perceptual sensitivity was associated with a deactivation of the left amygdala during the perceptual task. A consecutive psychophysiological interaction analysis showed increased connectivity of the amygdala with medial prefrontal cortex in the context of communicative compared to individual actions. Thus, whereas increased amygdala signaling might interfere with task-relevant processes, increased co-activation of the amygdala and the medial prefrontal cortex in a communicative context might represent the integration of mentalizing computations.

Social modulation of object-directed but not image-directed actions

There has recently been an increased research focus on the influence of social factors on human cognition, attention, and action. While this represents an important step towards an ecologically valid description of real-world behaviour, this work has primarily examined dyads interacting with virtual stimuli i.e. on-screen images of objects. Though differences between actions to images and real items are known, their relative sensitivity to social factors is largely unknown. We argue that because images and real items elicit different neural representations, patterns of attention, and hand actions, a direct comparison between the magnitude of social effects while interacting with images and real objects is demanded. We examined patterns of reaching as individuals performed a shape-matching game. Images and real objects were used as stimuli, and social context was manipulated via the proximity of an observer. We found that social context interacted with stimulus type to modulate behaviour. Specifically, there was a delay in reaching for distant objects when a participant was facing another individual but this social effect only occurred when the stimuli were real objects. Our data suggest that even when images and real objects are arranged to share the affordance of reachability, they differ in their sensitivity to social influences. Therefore, the measurement of social effects using on-screen stimuli may poorly predict the social effects of actions directed towards real objects. Accordingly, generalizations between these two domains should be treated with caution.

Visuo-motor interference with a virtual partner is equally present in cooperative and competitive interactions

Automatic imitation of observed actions is thought to be a powerful mechanism, one that may mediate the reward value of interpersonal interactions, but that could also generate visuo-motor interference when interactions involve complementary movements. Since interpersonal coordination seems to be crucial both when cooperating and competing with others, the questions arises as to whether imitation-and thus visuo-motor interference-occurs in both scenarios. To address this issue, we asked human participants to engage in high- or low-interactive (Interactive or Cued condition, respectively), cooperative or competitive, joint reach-to-grasps with a virtual partner. More specifically, interactions occurred in: (i) a Cued condition, where participants simply adapted their movement timing to synchronize with (during cooperation) or anticipate (during competition) the virtual partner's grasp; (ii) an Interactive condition requiring the same adaptation, as well as a real-time selection of their action according to the virtual character's movement. To simulate a realistic human-human interaction, the virtual character would change its movement speed in consecutive trials according to participants' behaviour. Results demonstrate that visuo-motor interference-as indexed by movement kinematics (higher maximum wrist height during complementary compared to imitative power grips)-emerge in both cooperative and competitive motor interactions only when predictions about the partner's movements are needed to perform one's own action (interactive condition). These results support the idea that simulative imitation is heavily present when individuals need to match their behaviours closely.

Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research

Action, gesture, and sign represent unique aspects of human communication that use form and movement to convey meaning. Researchers typically use manual coding of video data to characterize naturalistic, meaningful movements at various levels of description, but the availability of markerless motion-tracking technology allows for quantification of the kinematic features of gestures or any meaningful human movement. We present a novel protocol for extracting a set of kinematic features from movements recorded with Microsoft Kinect. Our protocol captures spatial and temporal features, such as height, velocity, submovements/strokes, and holds. This approach is based on studies of communicative actions and gestures and attempts to capture features that are consistently implicated as important kinematic aspects of communication. We provide open-source code for the protocol, a description of how the features are calculated, a validation of these features as quantified by our protocol versus manual coders, and a discussion of how the protocol can be applied. The protocol effectively quantifies kinematic features that are important in the production (e.g., characterizing different contexts) as well as the comprehension (e.g., used by addressees to understand intent and semantics) of manual acts. The protocol can also be integrated with qualitative analysis, allowing fast and objective demarcation of movement units, providing accurate coding even of complex movements. This can be useful to clinicians, as well as to researchers studying multimodal communication or human–robot interactions. By making this protocol available, we hope to provide a tool that can be applied to understanding meaningful movement characteristics in human communication.

Identifying others' informative intentions from movement kinematics

Previous research has demonstrated that people can reliably distinguish between actions with different instrumental intentions on the basis of the kinematic signatures of these actions (Cavallo, Koul, Ansuini, Capozzi, & Becchio, 2016). It has also been demonstrated that different informative intentions result in distinct action kinematics (McEllin, Knoblich, & Sebanz, 2017). However, it is unknown whether people can discriminate between instrumental actions and actions performed with an informative intention, and between actions performed with different informative intentions, on the basis of kinematic cues produced in these actions. We addressed these questions using a visual discrimination paradigm in which participants were presented with point light animations of an actor playing a virtual xylophone. We systematically manipulated and amplified kinematic parameters that have been shown to reflect different informative intentions. We found that participants reliably used both spatial and temporal cues in order to discriminate between instrumental actions and actions performed with an informative intention, and between actions performed with different informative intentions. Our findings indicate that the informative cues produced in joint action and teaching go beyond serving a general informative purpose and can be used to infer specific informative intentions.

Look at Me: Early Gaze Engagement Enhances Corticospinal Excitability During Action Observation

Direct gaze is a powerful social cue able to capture the onlooker’s attention. Beside gaze, head and limb movements as well can provide relevant sources of information for social interaction. This study investigated the joint role of direct gaze and hand gestures on onlookers corticospinal excitability (CE). In two experiments we manipulated the temporal and spatial aspects of observed gaze and hand behavior to assess their role in affecting motor preparation. To do this, transcranial magnetic stimulation (TMS) on the primary motor cortex (M1) coupled with electromyography (EMG) recording was used in two experiments. In the crucial manipulation, we showed to participants four video clips of an actor who initially displayed eye contact while starting a social request gesture, and then completed the action while directing his gaze toward a salient object for the interaction. This way, the observed gaze potentially expressed the intention to interact. Eye tracking data confirmed that gaze manipulation was effective in drawing observers’ attention to the actor’s hand gesture. In the attempt to reveal possible time-locked modulations, we tracked CE at the onset and offset of the request gesture. Neurophysiological results showed an early CE modulation when the actor was about to start the request gesture looking straight to the participants, compared to when his gaze was averted from the gesture. This effect was time-locked to the kinematics of the actor’s arm movement. Overall, data from the two experiments seem to indicate that the joint contribution of direct gaze and precocious kinematic information, gained while a request gesture is on the verge of beginning, increases the subjective experience of involvement and allows observers to prepare for an appropriate social interaction. On the contrary, the separation of gaze cues and body kinematics can have adverse effects on social motor preparation. CE is highly susceptible to biological cues, such as averted gaze, which is able to automatically capture and divert observer’s attention. This point to the existence of heuristics based on early action and gaze cues that would allow observers to interact appropriately.

Communicative intent modulates production and comprehension of actions and gestures: A Kinect study

Actions may be used to directly act on the world around us, or as a means of communication. Effective communication requires the addressee to recognize the act as being communicative. Humans are sensitive to ostensive communicative cues, such as direct eye gaze (Csibra & Gergely, 2009). However, there may be additional cues present in the action or gesture itself. Here we investigate features that characterize the initiation of a communicative interaction in both production and comprehension. We asked 40 participants to perform 31 pairs of object-directed actions and representational gestures in more- or less- communicative contexts. Data were collected using motion capture technology for kinematics and video recording for eye-gaze. With these data, we focused on two issues. First, if and how actions and gestures are systematically modulated when performed in a communicative context. Second, if observers exploit such kinematic information to classify an act as communicative. Our study showed that during production the communicative context modulates space-time dimensions of kinematics and elicits an increase in addressee-directed eye-gaze. Naïve participants detected communicative intent in actions and gestures preferentially using eye-gaze information, only utilizing kinematic information when eye-gaze was unavailable. Our study highlights the general communicative modulation of action and gesture kinematics during production but also shows that addressees only exploit this modulation to recognize communicative intention in the absence of eye-gaze. We discuss these findings in terms of distinctive but potentially overlapping functions of addressee directed eye-gaze and kinematic modulations within the wider context of human communication and learning.

When Height Carries Weight: Communicating Hidden Object Properties for Joint Action

In the absence of pre-established communicative conventions, people create novel communication systems to successfully coordinate their actions toward a joint goal. In this study, we address two types of such novel communication systems: sensorimotor communication, where the kinematics of instrumental actions are systematically modulated, versus symbolic communication. We ask which of the two systems co-actors preferentially create when aiming to communicate about hidden object properties such as weight. The results of three experiments consistently show that actors who knew the weight of an object transmitted this weight information to their uninformed co-actors by systematically modulating their instrumental actions, grasping objects of particular weights at particular heights. This preference for sensorimotor communication was reduced in a fourth experiment where co-actors could communicate with weight-related symbols. Our findings demonstrate that the use of sensorimotor communication extends beyond the communication of spatial locations to non-spatial, hidden object properties.

Abilities to Explicitly and Implicitly Infer Intentions from Actions in Adults with Autism Spectrum Disorder

Previous research suggests that Autism Spectrum Disorder (ASD) might be associated with impairments on implicit but not explicit mentalizing tasks. However, such comparisons are made difficult by the heterogeneity of stimuli and the techniques used to measure mentalizing capabilities. We tested the abilities of 34 individuals (17 with ASD) to derive intentions from others' actions during both explicit and implicit tasks and tracked their eye-movements. Adults with ASD displayed explicit but not implicit mentalizing deficits. Adults with ASD displayed typical fixation patterns during both implicit and explicit tasks. These results illustrate an explicit mentalizing deficit in adults with ASD, which cannot be attributed to differences in fixation patterns.

Vitality Forms Expressed by Others Modulate Our Own Motor Response: A Kinematic Study

During social interaction, actions, and words may be expressed in different ways, for example, gently or rudely. A handshake can be gentle or vigorous and, similarly, tone of voice can be pleasant or rude. These aspects of social communication have been named vitality forms by Daniel Stern. Vitality forms represent how an action is performed and characterize all human interactions. In spite of their importance in social life, to date it is not clear whether the vitality forms expressed by the agent can influence the execution of a subsequent action performed by the receiver. To shed light on this matter, in the present study we carried out a kinematic study aiming to assess whether and how visual and auditory properties of vitality forms expressed by others influenced the motor response of participants. In particular, participants were presented with video-clips showing a male and a female actor performing a "giving request" (give me) or a "taking request" (take it) in visual, auditory, and mixed modalities (visual and auditory). Most importantly, requests were expressed with rude or gentle vitality forms. After the actor's request, participants performed a subsequent action. Results showed that vitality forms expressed by the actors influenced the kinematic parameters of the participants' actions regardless to the modality by which they are conveyed.

Predictive Mechanisms Are Not Involved the Same Way during Human-Human vs. Human-Machine Interactions: A Review

Nowadays, interactions with others do not only involve human peers but also automated systems. Many studies suggest that the motor predictive systems that are engaged during action execution are also involved during joint actions with peers and during other human generated action observation. Indeed, the comparator model hypothesis suggests that the comparison between a predicted state and an estimated real state enables motor control, and by a similar functioning, understanding and anticipating observed actions. Such a mechanism allows making predictions about an ongoing action, and is essential to action regulation, especially during joint actions with peers. Interestingly, the same comparison process has been shown to be involved in the construction of an individual's sense of agency, both for self-generated and observed other human generated actions. However, the implication of such predictive mechanisms during interactions with machines is not consensual, probably due to the high heterogeneousness of the automata used in the experimentations, from very simplistic devices to full humanoid robots. The discrepancies that are observed during human/machine interactions could arise from the absence of action/observation matching abilities when interacting with traditional low-level automata. Consistently, the difficulties to build a joint agency with this kind of machines could stem from the same problem. In this context, we aim to review the studies investigating predictive mechanisms during social interactions with humans and with automated artificial systems. We will start by presenting human data that show the involvement of predictions in action control and in the sense of agency during social interactions. Thereafter, we will confront this literature with data from the robotic field. Finally, we will address the upcoming issues in the field of robotics related to automated systems aimed at acting as collaborative agents.

Cooperative and competitive contexts do not modify the effect of social intention on motor action

In social interactions, the movements performed by others can be used to anticipate their intention. The present paper investigates whether cooperative vs competitive contexts influence the kinematics of object-directed motor actions and whether they modulate the effect of social intention on motor actions. An "Actor" and a "Partner" participated in a task consisting in displacing a wooden dowel under time constraint. Before this Main action, the Actor performed a Preparatory action which consisted in placing the dowel at the center of the table. Information about who would make the forthcoming Main action was provided only to the Actor through headphones. Results demonstrate an exaggeration of spatial and temporal actions' parameters when acting for the Partner, in cooperative, as well as in competitive context. This finding suggests that the motor manifestation of social intention is largely determined by non-conscious implicit processes that seem little influenced by the context of social interaction.

Performance Monitoring Applied to System Supervision

Nowadays, automation is present in every aspect of our daily life and has some benefits. Nonetheless, empirical data suggest that traditional automation has many negative performance and safety consequences as it changed task performers into task supervisors. In this context, we propose to use recent insights into the anatomical and neurophysiological substrates of action monitoring in humans, to help further characterize performance monitoring during system supervision. Error monitoring is critical for humans to learn from the consequences of their actions. A wide variety of studies have shown that the error monitoring system is involved not only in our own errors, but also in the errors of others. We hypothesize that the neurobiological correlates of the self-performance monitoring activity can be applied to system supervision. At a larger scale, a better understanding of system supervision may allow its negative effects to be anticipated or even countered. This review is divided into three main parts. First, we assess the neurophysiological correlates of self-performance monitoring and their characteristics during error execution. Then, we extend these results to include performance monitoring and error observation of others or of systems. Finally, we provide further directions in the study of system supervision and assess the limits preventing us from studying a well-known phenomenon: the Out-Of-the-Loop (OOL) performance problem.

The role of perspective in discriminating between social and non-social intentions from reach-to-grasp kinematics

Making correct inferences regarding social and individual intentions may be crucial for successful interactions, especially when we are required to discriminate between cooperative and competitive behaviors. The results of previous studies indicate that reach-to-grasp kinematic parameters may be used to infer the social or individual outcome of a movement. However, the majority of the studies investigated this ability by presenting reach-to-grasp movements from a third-person perspective only. The aim of the present study was to assess whether the ability to recognize the intent associated to a reach-to-grasp movement varies as a function of perspective by manipulating the perspective of observation (second- and third-perspective) within participants. To this end, we presented participants with video clips of models performing a reach-to-grasp movement with different intents. The video clips were recorded both from a lateral view (third-person perspective) and from a frontal view (second-person perspective). After viewing the clips, in two subsequent tasks participants were asked to distinguish between social and non-social intentions by observing the initial phase of the same action recorded from the two different views. Results showed that, when a fast-speed movement was presented from a lateral view, participants were able to predict its social intention. In contrast, when the same movement was observed from a frontal view, performance was impaired. These results indicate that the ability to detect social intentions from motor cues can be biased by the visual perspective of the observer, specifically for fast-speed movements.

Sensorimotor Coarticulation in the Execution and Recognition of Intentional Actions

Humans excel at recognizing (or inferring) another's distal intentions, and recent experiments suggest that this may be possible using only subtle kinematic cues elicited during early phases of movement. Still, the cognitive and computational mechanisms underlying the recognition of intentional (sequential) actions are incompletely known and it is unclear whether kinematic cues alone are sufficient for this task, or if it instead requires additional mechanisms (e.g., prior information) that may be more difficult to fully characterize in empirical studies. Here we present a computationally-guided analysis of the execution and recognition of intentional actions that is rooted in theories of motor control and the coarticulation of sequential actions. In our simulations, when a performer agent coarticulates two successive actions in an action sequence (e.g., "reach-to-grasp" a bottle and "grasp-to-pour"), he automatically produces kinematic cues that an observer agent can reliably use to recognize the performer's intention early on, during the execution of the first part of the sequence. This analysis lends computational-level support for the idea that kinematic cues may be sufficiently informative for early intention recognition. Furthermore, it suggests that the social benefits of coarticulation may be a byproduct of a fundamental imperative to optimize sequential actions. Finally, we discuss possible ways a performer agent may combine automatic (coarticulation) and strategic (signaling) ways to facilitate, or hinder, an observer's action recognition processes.

Decoding intentions from movement kinematics

How do we understand the intentions of other people? There has been a longstanding controversy over whether it is possible to understand others' intentions by simply observing their movements. Here, we show that indeed movement kinematics can form the basis for intention detection. By combining kinematics and psychophysical methods with classification and regression tree (CART) modeling, we found that observers utilized a subset of discriminant kinematic features over the total kinematic pattern in order to detect intention from observation of simple motor acts. Intention discriminability covaried with movement kinematics on a trial-by-trial basis, and was directly related to the expression of discriminative features in the observed movements. These findings demonstrate a definable and measurable relationship between the specific features of observed movements and the ability to discriminate intention, providing quantitative evidence of the significance of movement kinematics for anticipating others' intentional actions.

Doing It Your Way: How Individual Movement Styles Affect Action Prediction

Individuals show significant variations in performing a motor act. Previous studies in the action observation literature have largely ignored this ubiquitous, if often unwanted, characteristic of motor performance, assuming movement patterns to be highly similar across repetitions and individuals. In the present study, we examined the possibility that individual variations in motor style directly influence the ability to understand and predict others' actions. To this end, we first recorded grasping movements performed with different intents and used a two-step cluster analysis to identify quantitatively 'clusters' of movements performed with similar movement styles (Experiment 1). Next, using videos of the same movements, we proceeded to examine the influence of these styles on the ability to judge intention from action observation (Experiments 2 and 3). We found that motor styles directly influenced observers' ability to 'read' others' intention, with some styles always being less 'readable' than others. These results provide experimental support for the significance of motor variability for action prediction, suggesting that the ability to predict what another person is likely to do next directly depends on her individual movement style.

Humans are sensitive to attention control when predicting others' actions

Studies of social perception report acute human sensitivity to where another's attention is aimed. Here we ask whether humans are also sensitive to how the other's attention is deployed. Observers viewed videos of actors reaching to targets without knowing that those actors were sometimes choosing to reach to one of the targets (endogenous control) and sometimes being directed to reach to one of the targets (exogenous control). Experiments 1 and 2 showed that observers could respond more rapidly when actors chose where to reach, yet were at chance when guessing whether the reach was chosen or directed. This implicit sensitivity to attention control held when either actor's faces or limbs were masked (experiment 3) and when only the earliest actor's movements were visible (experiment 4). Individual differences in sensitivity to choice correlated with an independent measure of social aptitude. We conclude that humans are sensitive to attention control through an implicit kinematic process linked to empathy. The findings support the hypothesis that social cognition involves the predictive modeling of others' attentional states.

Facilitation effect of observed motor deviants in a cooperative motor task: Evidence for direct perception of social intention in action

Spatiotemporal parameters of voluntary motor action may help optimize human social interactions. Yet it is unknown whether individuals performing a cooperative task spontaneously perceive subtly informative social cues emerging through voluntary actions. In the present study, an auditory cue was provided through headphones to an actor and a partner who faced each other. Depending on the pitch of the auditory cue, either the actor or the partner were required to grasp and move a wooden dowel under time constraints from a central to a lateral position. Before this main action, the actor performed a preparatory action under no time constraint, consisting in placing the wooden dowel on the central location when receiving either a neutral ("prêt"-ready) or an informative auditory cue relative to who will be asked to perform the main action (the actor: "moi"-me, or the partner: "lui"-him). Although the task focused on the main action, analysis of motor performances revealed that actors performed the preparatory action with longer reaction times and higher trajectories when informed that the partner would be performing the main action. In this same condition, partners executed the main actions with shorter reaction times and lower velocities, despite having received no previous informative cues. These results demonstrate that the mere observation of socially driven motor actions spontaneously influences the low-level kinematics of voluntary motor actions performed by the observer during a cooperative motor task. These findings indicate that social intention can be anticipated from the mere observation of action patterns.

Investigating the ability to read others' intentions using humanoid robots

The ability to interact with other people hinges crucially on the possibility to anticipate how their actions would unfold. Recent evidence suggests that a similar skill may be grounded on the fact that we perform an action differently if different intentions lead it. Human observers can detect these differences and use them to predict the purpose leading the action. Although intention reading from movement observation is receiving a growing interest in research, the currently applied experimental paradigms have important limitations. Here, we describe a new approach to study intention understanding that takes advantage of robots, and especially of humanoid robots. We posit that this choice may overcome the drawbacks of previous methods, by guaranteeing the ideal trade-off between controllability and naturalness of the interactive scenario. Robots indeed can establish an interaction in a controlled manner, while sharing the same action space and exhibiting contingent behaviors. To conclude, we discuss the advantages of this research strategy and the aspects to be taken in consideration when attempting to define which human (and robot) motion features allow for intention reading during social interactive tasks.