The role of shared visual information for joint action coordination

When "more for others, less for self" leads to co-benefits: A tri-MRI dyad-hyperscanning study

Unselfishness is admired, especially when collaborations between groups of various scales are urgently needed. However, its neural mechanisms remain elusive. In a tri-MRI dyad-hyperscanning experiment involving 26 groups, each containing 4 participants as two rotating pairs in a coordination game, we sought to achieve reciprocity, or "winning in turn by the two interacting players," as the precursor to unselfishness. Due to its critical role in social processing, the right temporal-parietal junction (rTPJ) was the seed for both time domain (connectivity) and frequency domain (i.e., coherence) analyses. For the former, negative connectivity between the rTPJ and the mentalizing network areas (e.g., the right inferior parietal lobule, rIPL) was identified, and such connectivity was further negatively correlated with the individual's final gain, supporting our task design that "rewarded" the reciprocal participants. For the latter, cerebral coherences of the rTPJs emerged between the interacting pairs (i.e., within-group interacting pairs), and the coupling between the rTPJ and the right superior temporal gyrus (rSTG) between the players who were not interacting with each other (i.e., within-group noninteracting pairs). These coherences reinforce the hypotheses that the rTPJ-rTPJ coupling tracks the collaboration processes and the rTPJ-rSTG coupling for the emergence of decontextualized shared meaning. Our results underpin two social roles (inferring others' behavior and interpreting social outcomes) subserved by the rTPJ-related network and highlight its interaction with other-self/other-concerning brain areas in reaching co-benefits among unselfish players.

Communicative modulations of early action components support the prediction of distal goals

The successful unfolding of many social interactions relies on our capacity to predict other people's action goals, whether these are proximal (i.e., immediate) or distal (i.e., upcoming). The present set of studies asks whether observers can predict the distal goal of two-step action sequences when presented with communicative modulations of the first movement component of the sequence. We conducted three online experiments in which we presented participants with animations of a box moving to a first target location before moving onwards to a final, either near or far, target location. The second movement component and the target locations were occluded. After observing the first movement, participants were asked to select the most likely final target location, i.e., the distal goal of the sequence. Experiment 1 showed that participants relied on the velocity modulations of the first movement to infer the distal goal. The results of Experiment 2 indicated that such predictions of distal goals are possible even when the second movement in the sequence does not contain any velocity information, thus suggesting that the information present in the first movement plays the major role in the process of linking movements to their distal goals. However, Experiment 3 showed that under some circumstances the second movement can also contribute to how observers predict a distal goal. We discuss these results in terms of the underlying simulation processes that enable observers to predict a distal goal from the observation of proximal communicative modulations.

The contribution of eye gaze and movement kinematics to the expression and identification of social intention in object-directed motor actions

The intention to include another person in an interaction (i.e., social intention) is known to influence the spatio-temporal characteristics of motor performances. However, the interplay between these kinematic variations and the social cues provided by eye gaze has not been properly assessed yet. In the present study, we tested whether limiting the access to eye gaze altered the motor-related effects of social intention on motor performances. In a dyadic interaction, the agents' task was to displace a dummy glass to a new position with the intention to fill it themselves (personal intention) or having it filled by the observers facing them (social intention). The observers performed their action only when they were able to identify a social intention in agents' action. The task was performed while having access to observers' eye gaze or not, through the manipulation of an occluder. Results showed an effect of social intention on agents' motor performances, that induced an amplification of the kinematic spatio-temporal parameters. Such amplification was smaller when the observers' eye gaze was not available. In this latter condition, the identification of the social intention in the observed actions was impaired. Altogether, the results suggest that the presence of eye gaze cues contributes significantly to the success of social interaction, by facilitating the expression and the understanding of social intentions through the kinematics of object-directed actions.

Excitatory/inhibitory motor balance reflects individual differences during joint action coordination

Joint action (JA) is a continuous process of motor co-regulation based on the integration of contextual (top-down) and kinematic (bottom-up) cues from partners. The fine equilibrium between excitation and inhibition in sensorimotor circuits is, thus, central to such a dynamic process of action selection and execution. In a bimanual task adapted to become a unimanual JA task, the participant held a bottle (JA), while a confederate had to reach and unscrew either that bottle or another stabilized by a mechanical clamp (No_JA). Prior knowledge was manipulated in each trial such that the participant knew (K) or not (No_K) the target bottle in advance. Online transcranial magnetic stimulation (TMS) was administered at action-relevant landmarks to explore corticospinal excitability (CSE) and inhibition (cortical silent period [cSP]). CSE was modulated early on before the action started if prior information was available. In contrast, cSP modulation emerged later during the reaching action, regardless of prior information. These two indexes could thus reflect the concurrent elaboration of contextual priors (top-down) and the online sampling of partner's kinematic cues (bottom-up). Furthermore, participants selected either one of two possible behavioural strategies, preferring early or late force exertion on the bottle. One translates into a reduced risk of motor coordination failure and the other into reduced metabolic expenditure. Each strategy was characterised by a specific excitatory/inhibitory profile. In conclusion, the study of excitatory/inhibitory balance paves the way for the neurophysiological determination of individual differences in the combination of top-down and bottom-up processing during JA coordination.

Space-time mapping relationships in sensorimotor communication during asymmetric joint action

Background

Sensorimotor communication is frequently observed in complex joint actions and social interactions. However, it remains challenging to explore the cognitive foundations behind sensorimotor communication.

Methods

The present study extends previous research by introducing a single-person baseline condition and formulates two distinct categories of asymmetric joint action tasks: distance tasks and orientation tasks. This research investigates the action performance of 65 participants under various experimental conditions utilizing a 2 (cooperative intention: Coop, No-coop) × 2 (task characteristic: distance, orientation) × 4 (target: T1, T2, T3, T4) repeated-measures experimental design to investigate the cognitive mechanisms underlying sensorimotor communication between individuals.

Results

The results showed that (1) target key dwell time, motion time, total motion time, and maximum motion height in the Coop condition are more than in the No-coop condition. (2) In the distance task without cooperative intention, the dwell time of T4 is smaller than T1, T2, T3, and its variability of T1, T2, T3, and T4 were no different. In the distance task with cooperative intention, the dwell time and its variability of T1, T2, T3, and T4 displayed an increasing trend. (3) In the orientation task without cooperative intention, the dwell time of T1 is smaller than T2, T3, T4, and variability of the target keys T1, T2, T3, and T4 had no difference. In the orientation task with cooperative intention, the dwell time and variability of the target keys T1, T2, T3, and T4 had increasing trends.

Conclusions

Those findings underscore the importance of cooperative intention for sensorimotor communication. In the distance task with cooperative intention, message senders establish a mapping relationship characterized by "near-small, far-large" between the task distance and the individual's action characteristics through sensorimotor experience. In the orientation task with cooperative intention, message senders combined sensorimotor experience and verbal metaphors to establish a mapping relationship between task orientation and action characteristics, following the sequence of "left-up, right-up, left-down, right-down" to transmit the message to others.

Flexible Cultural Learning Through Action Coordination

The cultural transmission of technical know-how has proven vital to the success of our species. The broad diversity of learning contexts and social configurations, as well as the various kinds of coordinated interactions they involve, speaks to our capacity to flexibly adapt to and succeed in transmitting vital knowledge in various learning contexts. Although often recognized by ethnographers, the flexibility of cultural learning has so far received little attention in terms of cognitive mechanisms. We argue that a key feature of the flexibility of cultural learning is that both the models and learners recruit cognitive mechanisms of action coordination to modulate their behavior contingently on the behavior of their partner, generating a process of mutual adaptation supporting the successful transmission of technical skills in diverse and fluctuating learning environments. We propose that the study of cultural learning would benefit from the experimental methods, results, and insights of joint-action research and, complementarily, that the field of joint-action research could expand its scope by integrating a learning and cultural dimension. Bringing these two fields of research together promises to enrich our understanding of cultural learning, its contextual flexibility, and joint action coordination.

Physiological Support of Goal-Directed Activity in Human

The effectiveness of goal-directed human behavior and the processes underlying organization of such activity are the subjects of various biomedical studies. Here we review both classical and modern evidence on the fundamental principles of goal-directed human activity. Facts are presented about the basic mechanisms that ensure the effectiveness of goal-directed behavior and determine its physiological cost.

An Integrated Model of Collaborative Skill Acquisition: Anticipation, Control Tuning, and Role Adoption

We studied collaborative skill acquisition in a dynamic setting with the game Co-op Space Fortress. While gaining expertise, the majority of subjects became increasingly consistent in the role they adopted without being able to communicate. Moreover, they acted in anticipation of the future task state. We constructed a collaborative skill acquisition model in the cognitive architecture ACT-R that reproduced subject skill acquisition trajectory. It modeled role adoption through reinforcement learning and predictive processes through motion extrapolation and learned relevant control parameters using both a reinforcement learning procedure and a new to ACT-R supervised learning procedure. This is the first integrated cognitive model of collaborative skill acquisition and, as such, gives us valuable insights into the multiple cognitive processes that are involved in learning to collaborate.

Motor performance in joint action tasks: The impact of dyadic motive fit

In many daily situations, two or more individuals need to coordinate their actions to achieve a common goal and perform successfully. Past research on joint action has predominantly focused on the question of how such interactions are accomplished. Here we focus on the impact of inter-individual, trait-like differences to predict joint action performance. More specifically, we examined whether performance in a joint action task is moderated by the (in)congruence of individuals' motive dispositions. To this end, 27 dyads performed a joint action task in which they had to navigate a ball through a maze with each partner using a joystick and each being responsible for either moving the ball along the x-axis or the y-axis. As dependent measures, we analyzed dyads' performance (times and errors). As trait-like predictors, we assessed implicit and explicit motives by means of the Picture Story Exercise and the Unified Motive Scale, respectively. Linear regression modeling revealed that congruent explicit affiliation motives predict faster best times and that higher congruent implicit achievement motives are associated with reduced errors. Exploratory Response Surface Analyses yielded identical results for the affiliation motive. These findings provide initial evidence to suggest that interindividual differences and in motives as well as their fit are related to joint action performance. Future directions of this new paradigm and novel ways to analyze dyadic motive fits and their relation to joint action performance are discussed.

Action prediction modulates self-other integration in joint action

People often coordinate actions with others, requiring an adjustable amount of self-other integration between actor's and co-actor's actions. Previous research suggests that such self-other integration (indexed by the joint Simon effect) is enhanced by agent similarity of the co-actor (e.g., high in intentionality). In this study, we aimed to extend this line of research by testing whether experiencing agency over a co-actor's actions (vicarious agency) and/or action prediction strengthens the joint Simon effect. For this purpose, we manipulated experienced agency by varying the experienced control over a co-actor's actions (Experiment 1), and action prediction regarding the co-actor's actions (Experiment 2). Vicarious agency could effectively be induced, but did not modulate the size of the joint Simon effect. The joint Simon effect was decreased when the co-actor's actions were unpredictable (vs. predictable) during joint task performance. These findings suggest social agency can be induced and effectively measured in joint action. Action prediction can act as an effective agency cue modulating the amount of self-other integration in joint action.

Communication and action predictability: two complementary strategies for successful cooperation

Making one's actions predictable and communicating what one intends to do are two strategies to achieve interpersonal coordination. It is less clear whether these two strategies are mutually exclusive or whether they can be used in parallel. Here, we asked how the availability of communication channels affects the use of strategy to make one's actions predictable. In three experiments, we investigated how people reach joint decisions if they are not allowed to communicate at all (Experiment 1), allowed minimal reciprocal communication (Experiment 2), or allowed to use the full range of conventional communication (Experiment 3). We found that when participants were not allowed to communicate, coordination was achieved by increasing action predictability. When conventional communication was allowed, there were no attempts to increase action predictability. In the minimal reciprocal communication condition, successful pairs both increased action predictability and established a communication system. Overall, this study demonstrates that people are able to flexibly adapt to coordination challenges during joint decision making and that communication reduces behavioural constraints on joint action coordination.

People infer communicative action through an expectation for efficient communication

Humans often communicate using body movements like winks, waves, and nods. However, it is unclear how we identify when someone’s physical actions are communicative. Given people’s propensity to interpret each other’s behavior as aimed to produce changes in the world, we hypothesize that people expect communicative actions to efficiently reveal that they lack an external goal. Using computational models of goal inference, we predict that movements that are unlikely to be produced when acting towards the world and, in particular, repetitive ought to be seen as communicative. We find support for our account across a variety of paradigms, including graded acceptability tasks, forced-choice tasks, indirect prompts, and open-ended explanation tasks, in both market-integrated and non-market-integrated communities. Our work shows that the recognition of communicative action is grounded in an inferential process that stems from fundamental computations shared across different forms of action interpretation.

Humans can quickly infer when someone’s body movements are meant to be communicative. Here, the authors show that this capacity is underpinned by an expectation that communicative actions will efficiently reveal that they lack an external goal.

Humans utilize sensory evidence of others' intended action to make online decisions

We often acquire sensory information from another person's actions to make decisions on how to move, such as when walking through a crowded hallway. Past interactive decision-making research has focused on cognitive tasks that did not allow for sensory information exchange between humans prior to a decision. Here, we test the idea that humans accumulate sensory evidence of another person's intended action to decide their own movement. In a competitive sensorimotor task, we show that humans exploit time to accumulate sensory evidence of another's intended action and utilize this information to decide how to move. We captured this continuous interactive decision-making behaviour with a drift-diffusion model. Surprisingly, aligned with a 'paralysis-by-analysis' phenomenon, we found that humans often waited too long to accumulate sensory evidence and failed to make a decision. Understanding how humans engage in interactive and online decision-making has broad implications that spans sociology, athletics, interactive technology, and economics.

Addressing joint action challenges in HRI: Insights from psychology and philosophy

The vast expansion of research in human-robot interactions (HRI) these last decades has been accompanied by the design of increasingly skilled robots for engaging in joint actions with humans. However, these advances have encountered significant challenges to ensure fluent interactions and sustain human motivation through the different steps of joint action. After exploring current literature on joint action in HRI, leading to a more precise definition of these challenges, the present article proposes some perspectives borrowed from psychology and philosophy showing the key role of communication in human interactions. From mutual recognition between individuals to the expression of commitment and social expectations, we argue that communicative cues can facilitate coordination, prediction, and motivation in the context of joint action. The description of several notions thus suggests that some communicative capacities can be implemented in the context of joint action for HRI, leading to an integrated perspective of robotic communication.

Your action does matter to me: Examining the role of the co-actor's action-effects in resolving the self-other discrimination problem

Sharing a task with another person can introduce the need to discriminate representations that refer to our own action from that of the other person's. The current understanding is that information about the stimulus event drives the self-other discrimination process, as it promotes (via the reactivation of feature codes) the representation that encodes the corresponding action. However, this mechanistic explanation relies on experimental situations in which stimulus event information (e.g., spatial location) is always and directly available. Thus, it remains unclear whether and how we could successfully discriminate between self- and other related action representations in the absence of such information. The present study addressed this unanswered question using a novel joint Simon task-based paradigm. We report the results of three experiments in which we manipulated the availability of stimulus event information into the contralateral space. Our findings demonstrate that participants are able to compensate for the absence of stimulus event information by relying on temporal features of their co-actor's action-effects (Experiment 1). Even more surprising was that participants continued to monitor the temporal features of their co-actor's actions even when given a verbal signal by their co-actor (Experiments 2a), or full access to the common workspace (Experiment 2b). Our results are strong evidence that the representation of actions is not purely stimulus driven. They suggest that the temporal dimension of the other person's actions is able to drive the self-other discrimination process, in the same way as other perceptual dimensions and feature codes that are shared with the stimulus event.

Multivariate model for cooperation: bridging social physiological compliance and hyperscanning

The neurophysiological analysis of cooperation has evolved over the past 20 years, moving towards the research of common patterns in neurophysiological signals of people interacting. Social physiological compliance (SPC) and hyperscanning represent two frameworks for the joint analysis of autonomic and brain signals, respectively. Each of the two approaches allows to know about a single layer of cooperation according to the nature of these signals: SPC provides information mainly related to emotions, and hyperscanning that related to cognitive aspects. In this work, after the analysis of the state of the art of SPC and hyperscanning, we explored the possibility to unify the two approaches creating a complete neurophysiological model for cooperation considering both affective and cognitive mechanisms We synchronously recorded electrodermal activity, cardiac and brain signals of 14 cooperative dyads. Time series from these signals were extracted, and multivariate Granger causality was computed. The results showed that only when subjects in a dyad cooperate there is a statistically significant causality between the multivariate variables representing each subject. Moreover, the entity of this statistical relationship correlates with the dyad’s performance. Finally, given the novelty of this approach and its exploratory nature, we provided its strengths and limitations.

The sound of silence: an EEG study of how musicians time pauses in individual and joint music performance

Pauses are an integral feature of social interaction. Conversation partners often pause between conversational turns, and musical co-performers often pause between musical phrases. How do humans coordinate the duration of pauses to ensure seamless interaction? A total of 40 trained pianists performed a simple melody containing fermatas (notated expressive pauses of unspecified duration) first alone (Solo) and then with a partner (Duet) while electroencephalography (EEG) was recorded. As predicted, Duet partners' tone onset synchrony was reduced for tones following pauses. Pauses were shorter in Duet relative to Solo performance, and synchrony of partners' Duet tone onsets was enhanced for tones following shorter pauses. EEG analysis revealed classic signatures of action preparation during pauses, namely decreases in the power of cortical beta oscillations (13-30 Hz, event-related desynchronization ERD). Beta ERD did not differ between pauses in Solo and Duet performance, but was enhanced for shorter relative to longer pauses, suggesting that reduced pause durations in Duet performance facilitated a neural state of enhanced action readiness. Together these findings provide novel insight into behavioural strategies by which musical partners resolve coordination challenges posed by expressive silence, and capture a clear neural signature of action planning during time-varying silences in natural music performance.

Mechanisms for mutual support in motor interactions

What is the key to successful interaction? Is it sufficient to represent a common goal, or does the way our partner achieves that goal count as well? How do we react when our partner misbehaves? We used a turn-taking music-like task requiring participants to play sequences of notes together with a partner, and we investigated how people adapt to a partner's error that violates their expectations. Errors consisted of either playing a wrong note of a sequence that the agents were playing together (thus preventing the achievement of the joint goal) or playing the expected note with an unexpected action. In both cases, we found post-error slowing and inaccuracy suggesting the participants' implicit tendency to correct the partner's error and produce the action that the partner should have done. We argue that these "joint" monitoring processes depend on the motor predictions made within a (dyadic) motor plan and may represent a basic mechanism for mutual support in motor interactions.

Progress in Joint-Action Research

Humans have a striking ability to coordinate their actions with each other to achieve joint goals. The tight interpersonal coordination that characterizes joint actions is achieved through processes that help with preparing for joint action as well as processes that are active while joint actions are being performed. To prepare for joint action, partners form representations of each other’s actions and tasks and the relation between them. This enables them to predict each other’s upcoming actions, which, in turn, facilitates coordination. While performing joint actions, partners’ coordination is maintained by (a) monitoring whether individual and joint outcomes correspond to what was planned, (b) predicting partners’ action parameters on the basis of familiarity with their individual actions, (c) communicating task-relevant information unknown to partners in an action-based fashion, and (d) relying on coupling of predictions through dense perceptual-information flow between coactors. The next challenge for the field of joint action is to generate an integrated perspective that links coordination mechanisms to normative, evolutionary, and communicative frameworks.

The influence of prior intention on joint action: an fNIRS-based hyperscanning study

Motor performances of the same action are affected by prior intentions to move unintentionally, cooperatively or competitively. Here, a back-and-forth movement task combined with a motion capture system and functional near-infrared spectroscopy (fNIRS)-based hyperscanning technology was utilized to record both the behavioral and neural data of 18 dyads of participants acting in pairs [joint conditions: no-intention, cooperative (Coop) and competitive (Comp)] or alone (single conditions: self-paced and fast-speed). The results revealed that Coop or Comp intentions in the joint conditions significantly sped up motor performance compared with similar single conditions, e.g. shorter movement times (MTs) in the Coop/Comp condition than the self-paced/fast-speed condition. Hemodynamic response analysis demonstrated that stronger activities for all joint conditions than the single conditions in the premotor and the supplementary motor cortex (Brodmann area 6) were independent of variations of MTs, indicating that they might reflect more complex aspects of action planning rather than simple execution-based processes. The comparisons of joint conditions across distinct prior intentions before acting yielded significant results for both behavioral and neural measures, with the highest activation of the temporo-parietal junction (TPJ) and the shortest MTs in the Comp condition considered to be implications for the top-down influence of prior intentions on joint performance.

Infants understand collaboration: Neural evidence for 9-month-olds' attribution of shared goals to coordinated joint actions

Interpreting others' actions as goal-directed, even when the actions are unfamiliar, is indispensable for social learning, and can be particularly important for infants, whose own action repertoire is limited. Indeed, young infants have been shown to attribute goals to unfamiliar actions as early as 3 months of age, but this ability appears restricted to actions performed by individuals. In contrast, attributing shared goals to actions performed by multiple individuals seems to emerge only in the second year of life. Considering the restrictions that this would impose on infants' understanding and learning from interactions in their environment, we reexamine this ability by introducing 9-month-old infants to simple joint actions, in which two agents coordinate their actions toward the same goal. To establish whether infants formed an expectation about future actions of these agents, infants' cortical activity was measured using functional near-infrared spectroscopy (fNIRS). The hemodynamic response, recorded in (p)STS, indicated that infants attributed goals to simultaneous and coordinated joint actions of two individuals. Thus, even prior to actively engaging in collaborative activities themselves, infants can attribute shared goals to observed joint actions, enabling infants to learn from, and about, the complementary roles of social interactions, a central characteristic of human culture.

How does a partner's motor variability affect joint action?

Motor learning studies demonstrate that an individual's natural motor variability predicts the rate at which she learns a motor task. Individuals exhibiting higher variability learn motor tasks faster, presumably because variability fosters exploration of a wider space of motor parameters. However, it is unclear how individuals regulate variability while learning a motor task together with a partner who perturbs their movements. In the current study, we investigated whether and how variability affects performance and learning in such joint actions. Participants learned to jointly perform a sequence of movements with a confederate who was either highly variable or less variable in her movements. A haptic coupling between the actors led to translation of partner's movement variability into a force perturbation. We tested how the variability and predictability of force perturbations coming from a partner foster or hamper individual and joint performance. In experiment 1, the confederate produced more or less variable range of force perturbations that occurred in an unpredictable order. In experiment 2, the confederate produced more or less variable force perturbations in a predictable order. In experiment 3, the confederate produced more or less variable force perturbations in which the magnitude of force delivered was predictable whereas the direction of the force was unpredictable. We analysed individual performance, measured as movement accuracy and joint performance, measured as interpersonal asynchrony. Results indicated that in all three experiments, participants successfully regulated the variability of their own movements. However, individual performance was worse when partner produced highly variable force perturbations in an unpredictable order. Interestingly, predictability of force perturbations offset the detrimental effects of variability on individual performance. Furthermore, participants in the high variability condition achieved higher flexibility and resilience for a wide range of force perturbations, when the partner produced predictable movements. Participants improved their joint performance with a highly variable partner only when the partner produced partially predictable movements. Our results indicate that individuals involved in a joint action selectively rely on either their own or their partner's variability (or both) for benefitting individual and joint action performance, depending on the predictability of the partner' movements.

Making oneself predictable in linguistic interactions

While language production is a highly demanding task, conversational partners are known to coordinate their turns with striking precision. Among the mechanisms that allow them to do so is listeners' ability to predict what the speaker will say, and thus to prepare their response in advance. But do speakers also play a role in facilitating coordination? We hypothesized that speakers contribute by using coordination smoothers - in particular by making their turns easier to predict. To test this, we asked participants to type definitions for common English words, either on their own (n = 26 individuals) or interacting with a partner (n = 18 pairs), and we measured the timing with which they produced the definitions. In a post-test, additional participants (n = 55) attempted to predict the final word of these definitions and rated them for quality. We found that interacting speakers initiated their turns with less variable delays than solo individuals. In contrast, our post-test measures suggested that jointly produced definitions were in fact of lower predictability and quality than those produced by individuals, but the analysis revealed these findings were likely confounded by task difficulty. We propose that the reduction in temporal variability observed for interacting speakers may facilitate prediction and thus act as a coordination smoother in linguistic interactions.

An approach to social flexibility: Congruency effects during spontaneous word-by-word interaction

Cognitive flexibility is the ability to switch between different concepts or to adapt goal-directed behavior in a changing environment. Although, cognitive research on this ability has long been focused on the individual mind, it is becoming increasingly clear that cognitive flexibility plays a central role in our social life. This is particularly evident in turn-taking in verbal conversation, where cognitive flexibility of the individual becomes part of social flexibility in the dyadic interaction. In this work, we introduce a model that reveals different parameters that explain how people flexibly handle unexpected events in verbal conversation. In order to study hypotheses derived from the model, we use a novel experimental approach in which thirty pairs of participants engaged in a word-by-word interaction by taking turns in generating sentences word by word. Similar to well established individual cognitive tasks, participants needed to adapt their behavior in order to respond to their co-actor’s last utterance. With our experimental approach we could manipulate the interaction between participants: Either both participants had to construct a sentence with a common target word (congruent condition) or with distinct target words (incongruent condition). We further studied the relation between the interactive Word-by-Word task measures and classical individual-centered, cognitive tasks, namely the Number-Letter task, the Stop-Signal task, and the GoNogo task. In the Word-by-Word task, we found that participants had faster response times in congruent compared to incongruent trials, which replicates the primary findings of standard cognitive tasks measuring cognitive flexibility. Further, we found a significant correlation between the performance in the Word-by-Word task and the Stop-Signal task indicating that participants with a high cognitive flexibility in the Word-by-Word task also showed high inhibition control.

Informing, Coordinating, and Performing: A Perspective on Functions of Sensorimotor Communication

Sensorimotor communication is a form of communication instantiated through body movements that are guided by both instrumental, goal-directed intentions and communicative, social intentions. Depending on the social interaction context, sensorimotor communication can serve different functions. This article aims to disentangle three of these functions: (a) an informing function of body movements, to highlight action intentions for an observer; (b) a coordinating function of body movements, to facilitate real-time action prediction in joint action; and (c) a performing function of body movements, to elicit emotional or aesthetic experiences in an audience. We provide examples of research addressing these different functions as well as some influencing factors, relating to individual differences, task characteristics, and situational demands. The article concludes by discussing the benefits of a closer dialog between separate lines of research on sensorimotor communication across different social contexts.

The combined effects of motor and social goals on the kinematics of object-directed motor action

Voluntary actions towards manipulable objects are usually performed with a particular motor goal (i.e., a task-specific object-target-effector interaction) and in a particular social context (i.e., who would benefit from these actions), but the mutual influence of these two constraints has not yet been properly studied. For this purpose, we asked participants to grasp an object and place it on either a small or large target in relation to Fitts’ law (motor goal). This first action prepared them for a second grasp-to-place action which was performed under temporal constraints, either by the participants themselves or by a confederate (social goal). Kinematic analysis of the first preparatory grasp-to-place action showed that, while deceleration time was impacted by the motor goal, peak velocity was influenced by the social goal. Movement duration and trajectory height were modulated by both goals, the effect of the social goal being attenuated by the effect of the motor goal. Overall, these results suggest that both motor and social constraints influence the characteristics of object-oriented actions, with effects that combine in a hierarchical way.

Emergence and suppression of cooperation by action visibility in transparent games

Real-world agents, humans as well as animals, observe each other during interactions and choose their own actions taking the partners' ongoing behaviour into account. Yet, classical game theory assumes that players act either strictly sequentially or strictly simultaneously without knowing each other's current choices. To account for action visibility and provide a more realistic model of interactions under time constraints, we introduce a new game-theoretic setting called transparent games, where each player has a certain probability of observing the partner's choice before deciding on its own action. By means of evolutionary simulations, we demonstrate that even a small probability of seeing the partner's choice before one's own decision substantially changes the evolutionary successful strategies. Action visibility enhances cooperation in an iterated coordination game, but reduces cooperation in a more competitive iterated Prisoner's Dilemma. In both games, "Win-stay, lose-shift" and "Tit-for-tat" strategies are predominant for moderate transparency, while a "Leader-Follower" strategy emerges for high transparency. Our results have implications for studies of human and animal social behaviour, especially for the analysis of dyadic and group interactions.

Instructions to attend to an observed action increase imitation in autistic adults

This study investigated whether reduced visual attention to an observed action might account for altered imitation in autistic adults. A total of 22 autistic and 22 non-autistic adults observed and then imitated videos of a hand producing sequences of movements that differed in vertical elevation while their hand and eye movements were recorded. Participants first performed a block of imitation trials with general instructions to imitate the action. They then performed a second block with explicit instructions to attend closely to the characteristics of the movement. Imitation was quantified according to how much participants modulated their movement between the different heights of the observed movements. In the general instruction condition, the autistic group modulated their movements significantly less compared to the non-autistic group. However, following instructions to attend to the movement, the autistic group showed equivalent imitation modulation to the non-autistic group. Eye movement recording showed that the autistic group spent significantly less time looking at the hand movement for both instruction conditions. These findings show that visual attention contributes to altered voluntary imitation in autistic individuals and have implications for therapies involving imitation as well as for autistic people's ability to understand the actions of others.

Reciprocal information flow and role distribution support joint action coordination

Many joint actions require task partners to temporally coordinate actions that follow different spatial patterns. This creates the need to find trade-offs between temporal coordination and spatial alignment. To study coordination under incongruent spatial and temporal demands, we devised a novel coordination task that required task partners to synchronize their actions while tracing different shapes that implied conflicting velocity profiles. In three experiments, we investigated whether coordination under incongruent demands is best achieved through mutually coupled predictions or through a clear role distribution with only one task partner adjusting to the other. Participants solved the task of trading off spatial and temporal coordination demands equally well when mutually perceiving each other’s actions without any role distribution, and when acting in a leader-follower configuration where the leader was unable to see the follower’s actions. Coordination was significantly worse when task partners who had been assigned roles could see each other’s actions. These findings make three contributions to our understanding of coordination mechanisms in joint action. First, they show that mutual prediction facilitates coordination under incongruent demands, demonstrating the importance of coupled predictive models in a wide range of coordination contexts. Second, they show that mutual alignment of velocity profiles in the absence of a leader-follower dynamic is more wide-spread than previously thought. Finally, they show that role distribution can result in equally effective coordination as mutual prediction without role assignment, provided that the role distribution is not arbitrarily imposed but determined by (lack of) perceptual access to a partner’s actions.

Oxytocin promotes action prediction

The neuropeptide oxytocin (OT) has been suggested to facilitate social cognition and behavior. As predicting others' behavior is at the core of human social-cognitive abilities and is indispensable for successful social interaction, we hypothesized that OT would increase action prediction. To test this hypothesis, 61 male and female healthy participants self-administered OT or placebo intranasally and their anticipatory eye-movements were recorded using eye-tracking techniques. We found that the ability to predict others' future actions was enhanced following OT treatment. This effect was mediated by the time to the first anticipatory eye-movement suggesting that improved action prediction might operate by increased attention to social cues. These findings provide direct evidence for the role of OT in promoting perception and processing of social cues.

Solo versus joint bimanual coordination

Understanding the differences between solo and joint action control is an important goal in psychology. The present study represented a novel approach in which participants performed a bimanual finger oscillation task, either alone or in pairs. It was hypothesized that performance of this task relies heavily on attention and utilizes two independent processes that differentially affect solo and joint performance. One process attempts to align the fingers correctly regardless of oscillation speed, and this is reflected in an alignment error evident even at slow oscillations. A second process attempts to minimize the time lag between the fingers as the oscillation speed increases, reflected in a temporal error indexed by the rate of error increase with increasing movement speed. In three experiments, alignment and temporal error in the finger oscillation task were compared in solo and joint actors. Overall, solo actors had much lower alignment error than joint actors. Solo actors also showed a reduction in temporal error when the fingers moved in a symmetrical rather than parallel fashion, consistent with previous research showing an increase in error with increasing movement speed. However, the effect of symmetry on temporal error did not occur with joint actors. Similar results were found with one hand inverted, suggesting that the pattern of results was not due to the use of homologous muscles. To test the role of visual feedback, we examined the effect of denying visual feedback to one of the actors in the joint condition. Paradoxically, under these conditions, there was lower temporal error in the symmetrical condition. These results are interpreted in terms of the organization of solo versus joint actions and the control of bimanual tasks in general.

Contribution of Developmental Psychology to the Study of Social Interactions: Some Factors in Play, Joint Attention and Joint Action and Implications for Robotics

Children exchange information through multiple modalities, including verbal communication, gestures and social gaze and they gradually learn to plan their behavior and coordinate successfully with their partners. The development of joint attention and joint action, especially in the context of social play, provides rich opportunities for describing the characteristics of interactions that can lead to shared outcomes. In the present work, we argue that human-robot interactions (HRI) can benefit from these developmental studies, through influencing the human's perception and interpretation of the robot's behavior. We thus endeavor to describe some components that could be implemented in the robot to strengthen the feeling of dealing with a social agent, and therefore improve the success of collaborative tasks. Focusing in particular on motor precision, coordination, and anticipatory planning, we discuss the question of complexity in HRI. In the context of joint activities, we highlight the necessity of (1) considering multiple speech acts involving multimodal communication (both verbal and non-verbal signals), and (2) analyzing separately the forms and functions of communication. Finally, we examine some challenges related to robot competencies, such as the issue of language and symbol grounding, which might be tackled by bringing together expertise of researchers in developmental psychology and robotics.

The way others move can influence what we choose

Whether pointing at a menu item or rifling through a clothes rack, when we choose we often move. We investigated whether people's tendency to copy the movements of others could influence their choices. Participants saw pairs of pictures in private and indicated which one they preferred. They then entered a virtual art gallery and saw the same picture pairs in the presence of a virtual character. Having observed the virtual character point to indicate her preference with either a high or low movement trajectory, participants indicated their preference. There was either an anatomical (same movement, same choice) or spatial correspondence (same movement, different choice) between the participant's pictures and those of the virtual character. We found that participants copied the movement made by the virtual character rather than her action goal (i.e., her choice of picture). This resulted in a shift towards the virtual character's preferences in the anatomical condition but away from her preferences in the spatial condition. This effect was driven by the observation of the virtual character's high pointing movements. In a further experiment, we did not find any significant differences in imitation behaviour in autism, although autistic participants were less consistent in their choices. Our findings demonstrate that we are not only influenced by other's choices but also the types of movements others make to indicate those choices.

Co-actors represent the order of each other's actions

Previous research has shown that people represent each other’s tasks and actions when acting together. However, less is known about how co-actors represent each other’s action sequences. Here, we asked whether co-actors represent the order of each other’s actions within an action sequence, or whether they merely represent the intended end state of a joint action together with their own contribution. In the present study, two co-actors concurrently performed action sequences composed of two actions. We predicted that if co-actors represent the order of each other’s actions, they should experience interference when the order of their actions differs. Supporting this prediction, the results of six experiments consistently showed that co-actors moved more slowly when performing the same actions in a different order compared to performing the same actions in the same order. In line with findings from bimanual movement tasks, our results indicate that interference can arise due to differences in movement parameters and due to differences in the perceptual characteristics of movement goals. The present findings extend previous research on co-representation, providing evidence that people represent not only the elements of another’s task, but also their temporal structure.

Identifying others' informative intentions from movement kinematics

Previous research has demonstrated that people can reliably distinguish between actions with different instrumental intentions on the basis of the kinematic signatures of these actions (Cavallo, Koul, Ansuini, Capozzi, & Becchio, 2016). It has also been demonstrated that different informative intentions result in distinct action kinematics (McEllin, Knoblich, & Sebanz, 2017). However, it is unknown whether people can discriminate between instrumental actions and actions performed with an informative intention, and between actions performed with different informative intentions, on the basis of kinematic cues produced in these actions. We addressed these questions using a visual discrimination paradigm in which participants were presented with point light animations of an actor playing a virtual xylophone. We systematically manipulated and amplified kinematic parameters that have been shown to reflect different informative intentions. We found that participants reliably used both spatial and temporal cues in order to discriminate between instrumental actions and actions performed with an informative intention, and between actions performed with different informative intentions. Our findings indicate that the informative cues produced in joint action and teaching go beyond serving a general informative purpose and can be used to infer specific informative intentions.

When Height Carries Weight: Communicating Hidden Object Properties for Joint Action

In the absence of pre-established communicative conventions, people create novel communication systems to successfully coordinate their actions toward a joint goal. In this study, we address two types of such novel communication systems: sensorimotor communication, where the kinematics of instrumental actions are systematically modulated, versus symbolic communication. We ask which of the two systems co-actors preferentially create when aiming to communicate about hidden object properties such as weight. The results of three experiments consistently show that actors who knew the weight of an object transmitted this weight information to their uninformed co-actors by systematically modulating their instrumental actions, grasping objects of particular weights at particular heights. This preference for sensorimotor communication was reduced in a fourth experiment where co-actors could communicate with weight-related symbols. Our findings demonstrate that the use of sensorimotor communication extends beyond the communication of spatial locations to non-spatial, hidden object properties.

Moving higher and higher: imitators' movements are sensitive to observed trajectories regardless of action rationality

Humans sometimes perform actions which, at least superficially, appear suboptimal to the goal they are trying to achieve. Despite being able to identify these irrational actions from an early age, humans display a curious tendency to copy them. The current study recorded participants’ movements during an established imitation task and manipulated the rationality of the observed action in two ways. Participants observed videos of a model point to a series of targets with either a low, high or ‘superhigh’ trajectory either in the presence or absence of obstacles between her targets. The participants’ task was to watch which targets the model pointed to and then point to the same targets on the table in front of them. There were no obstacles between the participants’ targets. Firstly, we found that the peak height of participants’ movements between their targets was sensitive to the height of the model’s movements, even in the ‘superhigh’ condition where the model’s action was rated as irrational. Secondly, participants showed obstacle priming—the peak height of participants’ movements was higher after having observed the model move over obstacles to reach her targets, compared to when there were no obstacles between her targets. This suggests that participants code the environment of co-actors into their own motor programs, even when this compromises the efficiency of their own movements. We discuss the implications of our findings in terms of theories of imitation and obstacle priming.

Autistic traits affect interpersonal motor coordination by modulating strategic use of role-based behavior

Background

Despite the fact that deficits in social communication and interaction are at the core of Autism Spectrum Conditions (ASC), no study has yet tested individuals on a continuum from neurotypical development to autism in an on-line, cooperative, joint action task. In our study, we aimed to assess whether the degree of autistic traits affects participants' ability to modulate their motor behavior while interacting in a Joint Grasping task and according to their given role.

Methods

Sixteen pairs of adult participants played a cooperative social interactive game in which they had to synchronize their reach-to-grasp movements. Pairs were comprised of one ASC and one neurotypical with no cognitive disability. In alternate experimental blocks, one participant knew what action to perform (instructed role) while the other had to infer it from his/her partner’s action (adaptive role). When in the adaptive condition, participants were told to respond with an action that was either opposite or similar to their partner. Participants also played a non-social control game in which they had to synchronize with a non-biological stimulus.

Results

In the social interactive task, higher degree of autistic traits predicted less ability to modulate joint action according to one’s interactive role. In the non-social task, autistic traits did not predict differences in movement preparation and planning, thus ruling out the possibility that social interactive task results were due to basic motor or executive function difficulties. Furthermore, when participants played the non-social game, the higher their autistic traits, the more they were interfered by the non-biological stimulus.

Conclusions

Our study shows for the first time that high autistic traits predict a stereotypical interaction style when individuals are required to modulate their movements in order to coordinate with their partner according to their role in a joint action task. Specifically, the infrequent emergence of role-based motor behavior modulation during on-line motor cooperation in participants with high autistic traits sheds light on the numerous difficulties ASC have in nonverbal social interactions.

Electronic supplementary material

The online version of this article (doi:10.1186/s13229-017-0141-0) contains supplementary material, which is available to authorized users.

Two Trackers Are Better than One: Information about the Co-actor's Actions and Performance Scores Contribute to the Collective Benefit in a Joint Visuospatial Task

When humans collaborate, they often distribute task demands in order to reach a higher performance compared to performing the same task alone (i.e., a collective benefit). Here, we tested to what extent receiving information about the actions of a co-actor, performance scores, or receiving both types of information impacts the collective benefit in a collaborative multiple object tracking task. In a between-subject design, pairs of individuals jointly tracked a subset of target objects among several moving distractor objects on a computer screen for a 100 trials. At the end of a trial, pairs received performance scores (Experiment 1), information about their partner's target selections (Experiment 2), or both types of information (Experiment 3). In all experiments, the performance of the pair exceeded the individual performances and the simulated performance of two independent individuals combined. Initially, when receiving both types of information (Experiment 3), pairs achieved the highest performance and divided task demands most efficiently compared to the other two experiments. Over time, performances and the ability to divide task demands for pairs receiving a single type of information converged with those receiving both, suggesting that pairs' coordination strategies become equally effective over time across experiments. However, pairs' performances never reached a theoretical limit of performance in all experiments. For distributing task demands, members of a pair predominantly used a left-right division of labor strategy (i.e., the leftmost targets were tracked by one co-actor while the rightmost targets were tracked by the other co-actor). Overall, findings of the present study suggest that receiving information about actions of a co-actor, performance scores, or receiving both enables pairs to devise effective division of labor strategies in a collaborative visuospatial task. However, when pairs had both types of information available, the formation of division of labor strategies was facilitated, indicating that pairs benefited the most from having both types of information available (i.e., actions about the co-actor and performance scores). Findings are applicable to circumstances in which humans need to perform collaborative visuospatial tasks that are time-critical and/or only allow a very limited exchange of information between co-actors.

Vision adds to haptics when dyads perform a whole-body joint balance task

When two or more people aim to produce joint action outcomes they need to coordinate their individual actions in space and time. Successful joint action performance has been reported to depend, among others, on visual and somatosensory information provided to the joint actors. This study investigated whether and how the systematic manipulation of visual information modulates real-time joint action when dyads performed a whole-body joint balance task. To this end, we introduced the Joint Action Board (JAB) where partners guided a ball through a maze towards a virtual hole by jointly shifting their weight on the board under three visual conditions: (1) the Follower had neither visual access to the Leader nor to the maze; (2) the Follower had no visual access to the maze but to the Leader; (3) the Follower had full visual access to both the Leader and to the maze. Joint action performance was measured as completion time of the maze task; interpersonal coordination was examined by means of kinematic analyses of both partners' motor behaviour. We predicted that systematically adding visual to the available haptic information would result in a significant increase in joint performance and that Leaders would change their coordination behavior depending on these conditions. Results showed that adding visual information to haptics led to an increase in joint action performance in a Leader-Follower relationship in a joint balance task. In addition, interpersonal coordination behavior (i.e. sway range of motion, time-lag between partner's bodies etc.) changed dependent on the provided visual information between partners in the jointly executed task.

Spontaneous adaptation explains why people act faster when being imitated

The human ability to perform joint actions is often attributed to high-level cognitive processes. For example, the finding that action leaders act faster when imitated by their partners has been interpreted as evidence for anticipation of the other's actions (Pfister, Dignath, Hommel, & Kunde, 2013). In two experiments, we showed that a low-level mechanism can account for this finding. Action leaders were faster when imitated than when counterimitated, but only if they could observe their partner's actions (Exp. 1). Crucially, when due to our manipulation the partner's imitative actions became slower than the counterimitative actions, leaders also became slower when they were imitated, and faster when counterimitated (Exp. 2). Our results suggest that spontaneous temporal adaptation is a key mechanism in joint action tasks. We argue for a reconsideration of other phenomena that have traditionally been attributed solely to high-level processes.