Come together: human-avatar on-line interactions boost joint-action performance in apraxic patients

Tool mastering today - an interdisciplinary perspective

Tools have coined human life, living conditions, and culture. Recognizing the cognitive architecture underlying tool use would allow us to comprehend its evolution, development, and physiological basis. However, the cognitive underpinnings of tool mastering remain little understood in spite of long-time research in neuroscientific, psychological, behavioral and technological fields. Moreover, the recent transition of tool use to the digital domain poses new challenges for explaining the underlying processes. In this interdisciplinary review, we propose three building blocks of tool mastering: (A) perceptual and motor abilities integrate to tool manipulation knowledge, (B) perceptual and cognitive abilities to functional tool knowledge, and (C) motor and cognitive abilities to means-end knowledge about tool use. This framework allows for integrating and structuring research findings and theoretical assumptions regarding the functional architecture of tool mastering via behavior in humans and non-human primates, brain networks, as well as computational and robotic models. An interdisciplinary perspective also helps to identify open questions and to inspire innovative research approaches. The framework can be applied to studies on the transition from classical to modern, non-mechanical tools and from analogue to digital user-tool interactions in virtual reality, which come with increased functional opacity and sensorimotor decoupling between tool user, tool, and target. By working towards an integrative theory on the cognitive architecture of the use of tools and technological assistants, this review aims at stimulating future interdisciplinary research avenues.

Effects of the social context on the neurophysiological correlates of observed error monitoring

Monitoring the motor performance of others, including the correctness of their actions, is crucial for the human behavior. However, while performance (and error) monitoring of the own actions has been studied extensively at the neurophysiological level, the corresponding studies on monitoring of others' errors are scarce, especially for ecological actions. Moreover, the role of the context of the observed action has not been sufficiently explored. To fill this gap, the present study investigated electroencephalographic (EEG) indices of error monitoring during observation of images of interrupted reach-to-grasp actions in social (an object held in another person's hand) and non-social (an object placed on a table) contexts. Analysis in time- and time-frequency domain showed that, at the level of conscious error awareness, there were no effects of the social context (observed error positivity was present for erroneous actions in both contexts). However, the effects of the context were present at the level of hand image processing: observing erroneous actions in the non-social context was related to larger occipito-temporal N1 and theta activity, while in the social context this pattern was reversed, i.e., larger N1 and theta activity were present for the correct actions. These results suggest that, in case of easily predictable ecological actions, action correctness is processed as early as at the level of hand image perception, since the hand posture conveys information about the action (e.g., motor intention). The social context of actions might make the correct actions more salient, possibly through the saliency of the correctly achieved common goal.

The dopaminergic system supports flexible and rewarding dyadic motor interactive behaviour in Parkinson's Disease

Studies indicate that the dopaminergic system (DAS) supports individual flexible behaviour. While flexibility is quintessential to effective dyadic motor interactions, whether DAS mediates adaptations of one's own motor behaviour to that of a partner is not known. Here, we asked patients with Parkinson's Disease (PD) to synchronize their grasping movements with those of a virtual partner in conditions that did (Interactive) or did not (Cued) require to predict and adapt to its actions. PD performed the task during daily antiparkinsonian treatment ('On' condition) or after drug-withdrawal ('Off' condition). A group of healthy individuals also served as control group. In the Interactive condition, PDs performed better and found the interaction more enjoyable when in 'On' than in 'Off' condition. Crucially, PD performance in the 'On' condition did not differ from that of healthy controls. This pattern of results hints at the key role of the DAS in supporting the flexible adaptation of one's own actions to the partner's during motor interactions.

The performance monitoring system is attuned to others' actions during dyadic motor interactions

Interpersonal motor interactions require the simultaneous monitoring of one's own and one's partner's actions. To characterize how the action monitoring system tracks self and other behavior during synchronous interactions, we combined electroencephalography recordings and immersive virtual reality in two tasks where participants were asked to synchronize their actions with those of a virtual partner (VP). The two tasks differed in the features to be monitored: the Goal task required participants to predict and monitor the VP's reaching goal; the Spatial task required participants to predict and monitor the VP's reaching trajectory. In both tasks, the VP performed unexpected movement changes to which the participant needed to adapt. By extracting the neural activity locked to the detection of unexpected changes in the VP's action (other-monitoring) or to the participants' action-replanning (self-monitoring), we show that the monitoring system is more attuned to others' than to one's own actions. Additionally, distinctive neural responses to VP's unexpected goals and trajectory corrections were found: goal changes were reflected both in early fronto-central and later posterior neural responses while trajectory deviations were reflected only in later posterior responses. Altogether, our results indicate that the monitoring system adopts an inherent social mode to handle interpersonal motor interactions.

Electrophysiological correlates of action monitoring in brain-damaged patients: A systematic review

Action monitoring is crucial to the successful execution of an action and understanding the actions of others. It is often impaired due to brain lesions, in particular after stroke. This systematic review aims to map the literature on the neurophysiological correlates of action monitoring in patients with brain lesions. Eighteen studies were identified and divided into two groups: studies on monitoring of one's own actions and studies on monitoring of the actions of others. The first group included EEG studies on monitoring of self-performed erroneous and correct actions. Impaired error detection (decreased error-related negativity) was observed in patients with lesions in the performance-monitoring network, as compared to healthy controls. Less consistent results were shown for error positivity and behavioral error monitoring performance. The second group of studies on monitoring of others' actions reported decreased mu frequency suppression, impaired readiness potential in the affected hemisphere and decreased EEG indices of error observation (observed error positivity and theta power) in stroke patients. As a whole, these results indicate distinct patterns of impaired neurophysiological activity related to monitoring one's own versus others' actions in patients with brain lesions. EEG recordings of this dissociation in the same patients might be a useful index of motor recovery, and therefore, potentially also beneficial in rehabilitation protocols.

Brain Dynamics of Action Monitoring in Higher-Order Motor Control Disorders: The Case of Apraxia

Limb apraxia (LA) refers to a high-order motor disorder characterized by the inability to reproduce transitive actions on commands or after observation. Studies demonstrate that action observation and action execution activate the same networks in the human brain, and provides an onlooker’s motor system with appropriate cognitive, motor and sensory-motor cues to flexibly implementing action-sequences and gestures. Tellingly, the temporal dynamics of action monitoring has never been explored in people suffering from LA. To fill this gap, we studied the electro-cortical signatures of error observation in human participants suffering from acquired left-brain lesions with (LA+) and without (LA–) LA, and in a group of healthy controls (H). EEG was acquired while participants observed from a first-person perspective (1PP) an avatar performing correct or incorrect reach-to-grasp a glass action in an immersive-virtual environment. Alterations of typical EEG signatures of error observation in time (early error positivity; Pe) and time-frequency domain (theta band-power) were found reduced in LA+ compared with H. Connectivity analyses showed that LA+ exhibited a decreased theta phase synchronization of both the frontoparietal and frontofrontal network, compared with H and LA–. Moreover, linear regression analysis revealed that the severity of LA [test of upper LA (TULIA) scores] was predicted by mid-frontal error-related theta activity, suggesting a link between error monitoring capacity and apraxic phenotypes. These results provide novel neurophysiological evidence of altered neurophysiological dynamics of action monitoring in individuals with LA and shed light on the performance monitoring changes occurring in this disorder.

Midfrontal Theta tACS Facilitates Motor Coordination in Dyadic Human-Avatar Interactions

Synchronous interpersonal motor interactions require moment-to-moment prediction and proactive monitoring of the partner's actions. Neurophysiologically, this is highlighted by an enhancement of midfrontal theta (4-7 Hz) oscillations. In this study, we explored the causal role of midfrontal theta for interpersonal motor interactions using transcranial alternating current stimulation (tACS). We implemented a realistic human-avatar interaction task in immersive virtual reality where participants controlled a virtual arm and hand to press a button synchronously with a virtual partner. Participants completed the task while receiving EEG-informed theta (Experiment 1) or beta (control frequency, Experiment 2) tACS over the frontal midline, as well as sham stimulation as a control. Results showed that midfrontal theta tACS significantly improved behavioral performance (i.e., reduced interpersonal asynchrony) and participants' motor strategies (i.e., increased movement times and reduced RTs), whereas beta tACS had no effect on these measures. These results suggest that theta tACS over frontal areas facilitates action monitoring and motor abilities supporting interpersonal interactions.

Abstract concepts in interaction: the need of others when guessing abstract concepts smooths dyadic motor interactions

concepts (ACs, e.g. 'justice') are more complex compared with concrete concepts (CCs) (e.g. 'table'). Indeed, they do not possess a single object as a referent, they assemble quite heterogeneous members and they are more detached from exteroceptive and more grounded in interoceptive experience. Recent views have hypothesized that interpersonal communication is particularly crucial to acquire and use ACs. The current study investigates the reliance of ACs/CCs representation on interpersonal behaviour. We asked participants to perform a motor interaction task with two avatars who embodied two real confederates. Before and after the motor interaction task, the two confederates provided participants with hints in a concept guessing task associated with visual stimuli: one helped in guessing ACs and the other, CCs. A control study we performed both with the materials employed in the main experiment and with other materials, confirmed that associating verbal concepts with visual images was more difficult with ACs than with CCs. Consistently, the results of the main experiment showed that participants asked for more hints with ACs than CCs and were more synchronous when interacting with the avatar corresponding to the AC's confederate. The results highlight an important role of sociality in grounding ACs.

Inhibitory Theta Burst Stimulation Highlights the Role of Left aIPS and Right TPJ during Complementary and Imitative Human-Avatar Interactions in Cooperative and Competitive Scenarios

Competitive and cooperative interactions are based on anticipation or synchronization with the partner's actions. Both forms of interaction may either require performing imitative or complementary movements with respect to those performed by our partner. We explored how parietal regions involved in the control of imitative behavior (temporo-parietal junction, TPJ), goal coding and visuo-motor integration (anterior intraparietal sulcus, aIPS) contribute to the execution of imitative and complementary movements during cooperative and competitive interactions. To this aim, we delivered off-line non-invasive inhibitory brain stimulation to healthy individuals' left aIPS and right TPJ before they were asked to reach and grasp an object together with a virtual partner by either performing imitative or complementary interactions. In different blocks, participants were asked to compete or cooperate with the virtual partner that varied its behavior according to cooperative or competitive contexts. Left aIPS and right TPJ inhibition impaired individuals' performance (i.e., synchrony in cooperative task and anticipation in competition) during complementary and imitative interactions, respectively, in both cooperative and competitive contexts, indicating that aIPS and TPJ inhibition affects own-other action integration and action imitation (that are different in complementary vs imitative interactions) more than action synchronization or anticipation (that are different in cooperative vs competitive contexts).

Competence-based social status and implicit preference modulate the ability to coordinate during a joint grasping task

Studies indicate that social status influences people's social perceptions. Less information is available about whether induced social status influences dyadic coordination during motor interactions. To explore this issue, we designed a study in which two confederates obtained high or low competence-based status by playing a game together with the participant, while the participant always occupied the middle position of the hierarchy. Following this status-inducing phase, participants were engaged in a joint grasping task with the high- and low-status confederates in different sessions while behavioural (i.e., interpersonal asynchrony and movement start time) indexes were measured. Participants' performance in the task (i.e., level of interpersonal asynchrony) when interacting with the low-status partner was modulated by their preference for him. The lower participants' preference for a low- relative to a high-status confederate, the worse participants' performance when interacting with the low-status confederate. Our results show that participants' performance during motor interactions changes according to the social status of the interaction partner.

Interpersonal Motor Interactions Shape Multisensory Representations of the Peripersonal Space

This perspective review focuses on the proposal that predictive multisensory integration occurring in one's peripersonal space (PPS) supports individuals' ability to efficiently interact with others, and that integrating sensorimotor signals from the interacting partners leads to the emergence of a shared representation of the PPS. To support this proposal, we first introduce the features of body and PPS representations that are relevant for interpersonal motor interactions. Then, we highlight the role of action planning and execution on the dynamic expansion of the PPS. We continue by presenting evidence of PPS modulations after tool use and review studies suggesting that PPS expansions may be accounted for by Bayesian sensory filtering through predictive coding. In the central section, we describe how this conceptual framework can be used to explain the mechanisms through which the PPS may be modulated by the actions of our interaction partner, in order to facilitate interpersonal coordination. Last, we discuss how this proposal may support recent evidence concerning PPS rigidity in Autism Spectrum Disorder (ASD) and its possible relationship with ASD individuals' difficulties during interpersonal coordination. Future studies will need to clarify the mechanisms and neural underpinning of these dynamic, interpersonal modulations of the PPS.

Doing it Wrong: A Systematic Review on Electrocortical and Behavioral Correlates of Error Monitoring in Patients with Neurological Disorders

Detecting errors in one's own and other's actions is a crucial ability for learning and adapting behavior to everchanging, highly volatile environments. Studies in healthy people demonstrate that monitoring errors in one's own and others' actions are underpinned by specific neural systems that are dysfunctional in a variety of neurological disorders. In this review, we first briefly discuss the main findings concerning error detection and error awareness in healthy subjects, the current theoretical models, and the tasks usually applied to investigate these processes. Then, we report a systematic search for evidence of dysfunctional error monitoring among neurological populations (basal ganglia, neurodegenerative, white-matter diseases and acquired brain injury). In particular, we examine electrophysiological and behavioral evidence for specific alterations of error processing in neurological disorders. Error-related negativity (ERN) amplitude were reduced in most (although not all) neurological patient groups, whereas Positivity Error (Pe) amplitude appeared not to be affected in most patient groups. Also theta activity was reduced in some neurological groups, but consistent evidence on the oscillatory activity has not been provided thus far. Behaviorally, we did not observe relevant patterns of pronounced dysfunctional (post-) error processing. Finally, we discuss limitations of the existing literature, conclusive points, open questions and new possible methodological approaches for clinical studies.

The Human Dynamic Clamp Reveals the Fronto-Parietal Network Linking Real-Time Social Coordination and Cognition

How does the brain allow us to interact with others? Social neuroscience has already provided some answers to these questions but has tended to treat high-level, cognitive interpretations of social behavior separately from the sensorimotor mechanisms upon which they rely. The goal here is to identify the underlying neural processes and mechanisms linking sensorimotor coordination and intention attribution. We combine the human dynamic clamp, a novel paradigm for studyingrealistic social behavior, with high-resolution electroencephalography. The collection of humanness and intention attribution reports, kinematics, and neural data affords an opportunity to relate brain activity to the ongoing social behavior. Behavioral results demonstrate that sensorimotor coordination influences the judgments of cooperativeness and humanness. Analysis of brain dynamics reveals two distinct networks related to the integration of visuo-motor information from self and other which overlap over the right parietal region. Furthermore, judgment of humanness and cooperation of others modulate the functional connectivity between this right parietal hub and the prefrontal cortex. These results reveal how distributed neural dynamics integrates information from "low-level" sensorimotor mechanisms and "high-level" social cognition to support the realistic social behaviors that play out in real time during interactive scenarios.

Midline frontal and occipito-temporal activity during error monitoring in dyadic motor interactions

Discrepancies between sensory predictions and action outcome are at the base of error coding. However, these phenomena have mainly been studied focussing on individual performance. Here, we explored EEG responses to motor prediction errors during a human-avatar interaction and show that Theta/Alpha activity of the frontal error-monitoring system works in phase with activity of the occipito-temporal node of the action observation network. Our motor interaction paradigm required healthy individuals to synchronize their reach-to-grasp movements with those of a virtual partner in conditions that did (Interactive) or did not require (Cued) movement prediction and adaptation to the partner's actions. Crucially, in 30% of the trials the virtual partner suddenly and unpredictably changed its movement trajectory thereby violating the human participant's expectation. These changes elicited error-related neuromarkers (ERN/Pe - Theta/Alpha modulations) over fronto-central electrodes during the Interactive condition. Source localization and connectivity analyses showed that the frontal Theta/Alpha activity induced by violations of the expected interactive movements was in phase with occipito-temporal Theta/Alpha activity. These results expand current knowledge about the neural correlates of on-line interpersonal motor interactions linking the frontal error-monitoring system to visual, body motion-related, responses.

How Task Interactivity Shapes Action Observation

Action observation triggers imitation, a powerful mechanism permitting interpersonal coordination. Coordination, however, also occurs when the partners’ actions are nonimitative and physically incongruent. One influential theory postulates that this is achieved via top-down modulation of imitation exerted by prefrontal regions. Here, we rather argue that coordination depends on sharing a goal with the interacting partner: this shapes action observation, overriding involuntary imitation, through the predictive activity of the left ventral premotor cortex (lvPMc). During functional magnetic resonance imaging (fMRI), participants played music in turn with a virtual partner in interactive and noninteractive conditions requiring 50% of imitative/nonimitative responses. In a full-factorial design, both perceptual features and low-level motor requirements were kept constant throughout the experiment. Behaviorally, the interactive context minimized visuomotor interference due to the involuntary imitation of physically incongruent movements. This was paralleled by modulation of neural activity in the lvPMc, which was specifically recruited during the interactive task independently of the imitative/nonimitative nature of the social exchange. This lvPMc activity reflected the predictive decoding of the partner’s actions, as revealed by multivariate pattern analysis. This demonstrates that, during interactions, we process our partners’ behavior to prospectively infer their contribution to the shared goal achievement, generating motor predictions for cooperation beyond low-level imitation.

Interactor's body shape does not affect visuo-motor interference effects during motor coordination

The biological-tuning of the Action Observation Network is highly debated. A current open question relates to whether the morphological appearance (body shape) and/or the biological motion of the observed agent triggers action simulation processes. Motor simulation of the partner's action is critical for motor interactions, where two partners coordinate their actions in space and time. It supports interpersonal alignment and facilitates online coordination. However, motor simulation also leads to visuo-motor interference effects when people are required to coordinate with complementary actions, i.e. incongruent movements as compared to the observed ones. Movement kinematics of interactive partners allows us to capture their automatic tendency to simulate and imitate the partner's complementary movements. In an online reach-to-grasp task, we investigated whether visuo-motor interference effects, visible in the kinematics of complementary movements, are modulated by the visual presence of the interactor's body shape. We asked participants to interact with 1) a humanoid agent with a human-like body shape and with real human, biological, movement kinematics, or 2) a non-humanoid agent, which did not resemble the human body-shape but moved with the same real kinematics. Through the combination of inferential and Bayesian statistics, the results show no effect of interactor's body shape on visuo-motor interference in reaching and grasping kinematics during online motor coordination. We discuss the results and propose that the kinematics of the observed movements, during motor interactions, might be the key factor for visuo-motor interference to take place independently from the morphological appearance of the partner. This is particularly relevant in a technological society that constantly asks humans to interact with artificial agents.

Inhibition of left anterior intraparietal sulcus shows that mutual adjustment marks dyadic joint-actions in humans

Creating real-life dynamic contexts to study interactive behaviors is a fundamental challenge for the social neuroscience of interpersonal relations. Real synchronic interpersonal motor interactions involve online, inter-individual mutual adaptation (the ability to adapt one’s movements to those of another in order to achieve a shared goal). In order to study the contribution of the left anterior Intra Parietal Sulcus (aIPS) (i.e. a region supporting motor functions) to mutual adaptation, here, we combined a behavioral grasping task where pairs of participants synchronized their actions when performing mutually adaptive imitative and complementary movements, with the inhibition of activity of aIPS via non-invasive brain stimulation. This approach allowed us to investigate whether aIPS supports online complementary and imitative interactions. Behavioral results showed that inhibition of aIPS selectively impairs pair performance during complementary compared to imitative interactions. Notably, this effect depended on pairs’ mutual adaptation skills and was higher for pairs composed of participants who were less capable of adapting to each other. Thus, we provide the first causative evidence for a role of the left aIPS in supporting mutually adaptive interactions and show that the inhibition of the neural resources of one individual of a pair is compensated at the dyadic level.

Social Cognition through the Lens of Cognitive and Clinical Neuroscience

Social cognition refers to a set of processes, ranging from perception to decision-making, underlying the ability to decode others' intentions and behaviors to plan actions fitting with social and moral, besides individual and economic considerations. Its centrality in everyday life reflects the neural complexity of social processing and the ubiquity of social cognitive deficits in different pathological conditions. Social cognitive processes can be clustered in three domains associated with (a) perceptual processing of social information such as faces and emotional expressions (social perception), (b) grasping others' cognitive or affective states (social understanding), and (c) planning behaviors taking into consideration others', in addition to one's own, goals (social decision-making). We review these domains from the lens of cognitive neuroscience, i.e., in terms of the brain areas mediating the role of such processes in the ability to make sense of others' behavior and plan socially appropriate actions. The increasing evidence on the “social brain” obtained from healthy young individuals nowadays constitutes the baseline for detecting changes in social cognitive skills associated with physiological aging or pathological conditions. In the latter case, impairments in one or more of the abovementioned domains represent a prominent concern, or even a core facet, of neurological (e.g., acquired brain injury or neurodegenerative diseases), psychiatric (e.g., schizophrenia), and developmental (e.g., autism) disorders. To pave the way for the other papers of this issue, addressing the social cognitive deficits associated with severe acquired brain injury, we will briefly discuss the available evidence on the status of social cognition in normal aging and its breakdown in neurodegenerative disorders. Although the assessment and treatment of such impairments is a relatively novel sector in neurorehabilitation, the evidence summarized here strongly suggests that the development of remediation procedures for social cognitive skills will represent a future field of translational research in clinical neuroscience.

Visuo-motor interference with a virtual partner is equally present in cooperative and competitive interactions

Automatic imitation of observed actions is thought to be a powerful mechanism, one that may mediate the reward value of interpersonal interactions, but that could also generate visuo-motor interference when interactions involve complementary movements. Since interpersonal coordination seems to be crucial both when cooperating and competing with others, the questions arises as to whether imitation-and thus visuo-motor interference-occurs in both scenarios. To address this issue, we asked human participants to engage in high- or low-interactive (Interactive or Cued condition, respectively), cooperative or competitive, joint reach-to-grasps with a virtual partner. More specifically, interactions occurred in: (i) a Cued condition, where participants simply adapted their movement timing to synchronize with (during cooperation) or anticipate (during competition) the virtual partner's grasp; (ii) an Interactive condition requiring the same adaptation, as well as a real-time selection of their action according to the virtual character's movement. To simulate a realistic human-human interaction, the virtual character would change its movement speed in consecutive trials according to participants' behaviour. Results demonstrate that visuo-motor interference-as indexed by movement kinematics (higher maximum wrist height during complementary compared to imitative power grips)-emerge in both cooperative and competitive motor interactions only when predictions about the partner's movements are needed to perform one's own action (interactive condition). These results support the idea that simulative imitation is heavily present when individuals need to match their behaviours closely.

Distortion of Visuo-Motor Temporal Integration in Apraxia: Evidence From Delayed Visual Feedback Detection Tasks and Voxel-Based Lesion-Symptom Mapping

Limb apraxia is a higher brain dysfunction that typically occurs after left hemispheric stroke and its cause cannot be explained by sensory disturbance or motor paralysis. The comparison of motor signals and visual feedback to generate errors, i.e., visuo-motor integration, is important in motor control and motor learning, which may be impaired in apraxia. However, in apraxia after stroke, it is unknown whether there is a specific deficit in visuo-motor temporal integration compared to visuo-tactile and visuo-proprioceptive temporal integration. We examined the precision of visuo-motor temporal integration and sensory-sensory (visuo-tactile and visuo-proprioception) temporal integration in apraxia after stroke by using a delayed visual feedback detection task with three different conditions (tactile, passive movement, and active movement). The delay detection threshold and the probability curve for delay detection obtained in this task were quantitative indicators of the respective temporal integration functions. In addition, we performed subtraction and voxel-based lesion-symptom mapping to identify the brain lesions responsible for apraxia and deficits in visuo-motor temporal integration. The behavioral experiments showed that the delay detection threshold was extended and that the probability curve for delay detection was less steep in apraxic patients compared to controls (pseudo-apraxic patients and unaffected patients), only for the active movement condition, and not for the tactile and passive movement conditions. Furthermore, the severity of apraxia was significantly correlated with the delay detection threshold and the steepness of the probability curve in the active movement condition. These results indicated that multisensory (i.e., visual, tactile, and proprioception) feedback was normally temporally integrated, but motor prediction and visual feedback were not correctly temporally integrated in apraxic patients. That is, apraxic patients had difficulties with visuo-motor temporal integration. Lesion analyses revealed that both apraxia and the distortion of visuo-motor temporal integration were associated with lesions in the fronto-parietal motor network, including the left inferior parietal lobule and left inferior frontal gyrus. We suppose that damage to the left inferior fronto-parietal network could cause deficits in motor prediction for visuo-motor temporal integration, but not for sensory-sensory (visuo-tactile and visuo-proprioception) temporal integration, leading to the distortion of visuo-motor temporal integration in patients with apraxia.