Task representation in individual and joint settings

Joint action with human and robotic co-actors: Self-other integration is immune to the perceived humanness of the interacting partner

When performing a joint action task, we automatically represent the action and/or task constraints of the co-actor with whom we are interacting. Current models suggest that, not only physical similarity, but also abstract, conceptual features shared between self and the interacting partner play a key role in the emergence of joint action effects. Across two experiments, we investigated the influence of the perceived humanness of a robotic agent on the extent to which we integrate the action of that agent into our own action/task representation, as indexed by the Joint Simon Effect (JSE). The presence (vs. absence) of a prior verbal interaction was used to manipulate robot's perceived humanness. In Experiment 1, using a within-participant design, we had participants perform the joint Go/No-go Simon task with two different robots. Before performing the joint task, one robot engaged in a verbal interaction with the participant and the other robot did not. In Experiment 2, we employed a between-participants design to contrast these two robot conditions as well as a human partner condition. In both experiments, a significant Simon effect emerged during joint action and its amplitude was not modulated by the humanness of the interacting partner. Experiment 2 further showed that the JSE obtained in robot conditions did not differ from that measured in the human partner condition. These findings contradict current theories of joint action mechanisms according to which perceived self-other similarity is a crucial determinant of self-other integration in shared task settings.

Digitally embodied lifespan neurocognitive development and Tactile Internet: Transdisciplinary challenges and opportunities

Mechanisms underlying perceptual processing and inference undergo substantial changes across the lifespan. If utilized properly, technologies could support and buffer the relatively more limited neurocognitive functions in the still developing or aging brains. Over the past decade, a new type of digital communication infrastructure, known as the "Tactile Internet (TI)," is emerging in the fields of telecommunication, sensor and actuator technologies and machine learning. A key aim of the TI is to enable humans to experience and interact with remote and virtual environments through digitalized multimodal sensory signals that also include the haptic (tactile and kinesthetic) sense. Besides their applied focus, such technologies may offer new opportunities for the research tapping into mechanisms of digitally embodied perception and cognition as well as how they may differ across age cohorts. However, there are challenges in translating empirical findings and theories about neurocognitive mechanisms of perception and lifespan development into the day-to-day practices of engineering research and technological development. On the one hand, the capacity and efficiency of digital communication are affected by signal transmission noise according to Shannon's (1949) Information Theory. On the other hand, neurotransmitters, which have been postulated as means that regulate the signal-to-noise ratio of neural information processing (e.g., Servan-Schreiber et al., 1990), decline substantially during aging. Thus, here we highlight neuronal gain control of perceptual processing and perceptual inference to illustrate potential interfaces for developing age-adjusted technologies to enable plausible multisensory digital embodiments for perceptual and cognitive interactions in remote or virtual environments.

Gender composition of pairs influences joint action effect

Research on joint action has demonstrated that individuals are sensitive to a coactor's attentional relation to jointly attend stimuli. It has also been suggested that some features are necessary to resolve the discrimination problem (i.e., self-own and other-own actions). In the present study, we aimed to test whether the gender composition of interacting pairs modulated the joint action effect. Same- (female-female or male-male) and mixed- (female-male) gender pairs performed a joint version of flanker tasks in Experiment 1 (90 participants, 50% males), while in Experiment 2 (154 participants, 50% males) Navon tasks were performed. In Experiment 1, a higher joint flanker effect in same-gender pairs than in mixed-gender pairs, and this joint effect was similar to the classical flanker effect reported by males and females in a classical procedure of the task (70 participants, 50% males). In Experiment 2, the same-gender pairs reported a joint Navon effect, which was reversed in mixed-gender pairs. In conclusion, our findings support how the gender composition of interacting pairs plays a role in joint attentional tasks.

The role of action inhibition for behavioral control in joint action

When two individuals share a task with a common goal, coordinating one's own and the other's actions is pivotal. Inhibition of one's own actions when it is the other's turn to act is assumed to play a crucial role in this process. For instance, in the joint Simon task, two individuals share a two-choice task such that one of them responds to one stimulus type and ignores the stimulus type to which the other responds. Because stimuli can either appear on one's own or on the other's side, stimulus location can conflict with stimulus identity, thus slowing response time. It has previously been shown that such conflict leads to a reduction of the detrimental effects of conflict on immediately upcoming trials both following own responses and even more so following the other's responses. This amplified trial-to-trial adjustment following the other's responses has been assumed to reflect the inhibition of own responses on the other's trials. The present study tested this hypothesis by comparing sequential trial-to-trial adjustments following correct responses and commission errors on which the inhibition of own responses has failed. As expected, adjustments were stronger following the other's correct responses than following own correct responses. Crucially, such amplification of sequential adjustment was not observed following own commission errors on the other's trials. This shows that amplification of sequential adjustments following the other's trials depend on successful inhibition of own responses on these trials and points to a crucial role of response inhibition for behavioral control in joint action.

Understanding joint action: Current theoretical and empirical approaches

Joint actions are omnipresent, ranging from a handshake between two people to the coordination of groups of people playing in an orchestra. We are highly skilled at coordinating our actions with those of others to reach common goals and rely on this ability throughout our daily lives. What are the social, cognitive and neural processes underlying this ability? How do others around us influence our task representations? How does joint action influence interpersonal interactions? How do language and gesture support joint action? What differentiates joint action from individual action? This article forms an introductory editorial to the field of joint action. It accompanies contributions to the special issue entitled "Current Issues in Joint Action Research". The issue brings together conceptual and empirical approaches on different topics, ranging from lower-level issues such as the link between perception and joint action, to higher-level issues such as language as a form of joint action.

Metacontrol and joint action: how shared goals transfer from one task to another?

In most of our daily activities and in team sports, we interact with other individuals and do not act in isolation. Using a social variant of the standard two-choice Simon task, this study aims to test if competitive/cooperative processing modes (i.e., metacontrol states) change the degree of bodily self-other integration between two persons in joint action. In addition, and more exploratory the study tested if this effect depends on a shared group experience with the partner. Two participants shared a visual Simon task, so that each person basically performed complementary parts of the task, which transfers the paradigm into a go/no-go Simon task for each person. Before running this joint Simon task, we set both participants either in a competitive or a cooperative control state by means of a dyadic game, a manipulation aimed at testing possible goal transfer across tasks. We found significant joint Simon effects for participants who were in a competitive state and for participants who were in a cooperative state. The joint Simon effect for participants being in a competitive state was significantly smaller than for participants being in a cooperative state. When experiencing the goal induction together with the partner, the joint Simon effect was significantly decreased as when the induction was performed alone. Both effects (metacontrol state induction and shared experience) seem to be statistically independent of each other. In line with predictions of metacontrol state theory, our study indicated that abstract cognitive goal states can be transferred from one task to another task, able to affect the degree of bodily self-other integration across different task situations.

The role of the co-actor's response reachability in the joint Simon effect: remapping of working space by tool use

The Simon effect, that is the advantage of the spatial correspondence between stimulus and response locations when stimulus location is task irrelevant, occurs even when the task is performed by two participants, each performing a go/no-go task. This effect, known as the joint Simon effect, does not emerge when participants sit outside each other's peripersonal space, thus suggesting that the presence of an active confederate in peripersonal space might provide a reference for response coding. The present study investigated whether this finding is due to the distance separating the participants and/or to the distance separating each participant and the other agent's response. In two experiments, pairs of participants performed a social detection task sitting outside each other's arm reach, with response keys located close to the participants or outside arm reach. When the response key was located outside the participant's arm reach, he/she could reach it by means of a tool. In Experiment 1, by means of a tool, participants could reach their response key only, while in Experiment 2, they could reach also their co-agent's response key. The joint Simon effect did not emerge when participants could not reach the co-actor's response, while it emerged when they could potentially reach the other participant's response using the tool, but only when turn taking was required. These results may be taken as evidence that the possibility to reach and act upon the co-actor's response key may be at the bases of compatibility effects observed in joint action contexts requiring complementary responses.

Sharing Different Reference Frames: How Stimulus Setup and Task Setup Shape Egocentric and Allocentric Simon Effects

Different reference frames are used in daily life in order to structure the environment. The two-choice Simon task setting has been used to investigate how task-irrelevant spatial information influences human cognitive control. In recent studies, a Go/NoGo Simon task setting was used in order to divide the Simon task between a pair of participants. Yet, not only a human co-actor, but also even an attention-grabbing object can provide sufficient reference in order to reintroduce a Simon effect (SE) indicating cognitive conflict in Go/NoGo task settings. Interestingly, the SE could only occur when a reference point outside of the stimulus setup was available. The current studies exploited the dependency between different spatial reference frames (egocentric and allocentric) offered by the stimulus setup itself and the task setup (individual vs. joint Go/NoGot task setting). Two studies (Experiments 1 and 2) were carried out along with a human co-actor. Experiment 3 used an attention-grabbing object instead. The egocentric and allocentric SEs triggered by different features of the stimulus setup (global vs. local) were modulated by the task setup. When interacting with a human co-actor, an egocentric SE was found for global features of the stimulus setup (i.e., stimulus position on the screen). In contrast, an allocentric SE was yielded in the individual task setup illustrating the relevance of more local features of the stimulus setup (i.e., the manikin’s ball position). Results point toward salience shifts between different spatial reference frames depending on the nature of the task setup.

The Multimodal Go-Nogo Simon Effect: Signifying the Relevance of Stimulus Features in the Go-Nogo Simon Paradigm Impacts Event Representations and Task Performance

Numerous studies have shown that stimulus-response-compatibility (SRC) effects in the go-nogo version of the Simon task can be elicited as a result of performing the task together with another human or non-human agent (e.g., a Japanese-waving-cat, a working-clock, or a ticking-metronome). A parsimonious explanation for both social and non-social SRC effects is that highlighting the spatial significance of alternative (non-/social) action events makes action selection more difficult. This holds even when action events are task-irrelevant. Recent findings, however, suggest that this explanation holds only for cases of a modality correspondence between the Simon task as such (i.e., auditory or visual) and the alternative (non-/social) action event that needs to be discriminated. However, based on the fact that perception and action are represented by the same kind of codes, an event that makes the go-nogo decision more challenging should impact go-nogo Simon task performance. To tackle this issue, the present study tested if alternative stimulus events that come from a different sensory modality do impact SRC effects in the go-nogo version of the Simon task. This was tested in the presence and absence of alternative action events of a human co-actor. In a multimodal (auditory–visual) go-nogo Simon paradigm, participants responded to their assigned stimulus – e.g., a single auditory stimulus while ignoring the alternative visual stimulus or vice versa – in the presence or absence of a human co-actor (i.e., joint and single go-nogo condition). Results showed reliable SRCs in both, single and joint go-nogo Simon task conditions independent of the modality participants had to respond to. Although a correspondence between stimulus material and attention-grabbing event might be an efficient condition for SRCs to emerge, the driving force underlying the emergence of SRCs rather appears to be whether the attentional focus prevents or facilitates alternative events to be integrated. Thus, under task conditions in which the attentional focus is sufficiently broad to enable the integration and thus cognitive representation of alternative events, go-nogo decisions become more difficult, resulting in reliable SRCs in single and joint go-nogo Simon tasks.

Response Coordination Emerges in Cooperative but Not Competitive Joint Task

Effective social interactions rely on humans' ability to attune to others within social contexts. Recently, it has been proposed that the emergence of shared representations, as indexed by the Joint Simon effect (JSE), might result from interpersonal coordination (Malone et al., 2014). The present study aimed at examining interpersonal coordination in cooperative and competitive joint tasks. To this end, in two experiments we investigated response coordination, as reflected in instantaneous cross-correlation, when co-agents cooperate (Experiment 1) or compete against each other (Experiment 2). In both experiments, participants performed a go/no-go Simon task alone and together with another agent in two consecutive sessions. In line with previous studies, we found that social presence differently affected the JSE under cooperative and competitive instructions. Similarly, cooperation and competition were reflected in co-agents response coordination. For the cooperative session (Experiment 1), results showed higher percentage of interpersonal coordination for the joint condition, relative to when participants performed the task alone. No difference in the coordination of responses occurred between the individual and the joint conditions when co-agents were in competition (Experiment 2). Finally, results showed that interpersonal coordination between co-agents implies the emergence of the JSE. Taken together, our results suggest that shared representations seem to be a necessary, but not sufficient, condition for interpersonal coordination.

Is Your Color My Color? Dividing the Labor of the Stroop Task Between Co-actors

Performing a task with other actors involves two opposing forces, division of labor between co-acting individuals and integration of divided parts of the task into a shared mental representation (co-representation). Previous studies have focused primarily on the integration of task representations and limited attention has paid to the division of labor. The present study devised a test of the integration and the division in a joint task setting. A joint version of the Stroop task was developed, in which pairs of actors were assigned different sets of target colors. If the actors integrate their co-actor’s task, the colors assigned to their co-actor should be represented as if they were the actor’s own target colors; the Stroop effect should be as large when distractor color words denote their co-actor’s target colors as when these words denote the actor’s own target colors. If the actors divide the labor of the Stroop task, the colors assigned to their partner should be represented as non-target colors; the Stroop effect should be smaller when the distractor color words denote the co-actor’s target colors than when these words denote the actor’s own target colors. The results of response time did not provide clear support for either position, while those of response accuracy supported the division of labor. Possible cognitive mechanisms that support the division of labor and the integration of task representation are discussed.

Negotiating between individual and joint goals in ensemble musical performance

Successful joint action requires negotiation, especially in the event of goal incongruence. This article addresses goal incongruence in joint musical performance by manipulating the congruence of score instructions (congruent/incongruent) regarding tempo (speed) and dynamics (sound intensity) given to piano duos. The aim is to investigate how co-performers negotiate incongruent instructions for tempo and dynamics by balancing the prioritisation of individual goals versus the joint outcome and how this negotiation is modulated by musical expertise and personality (locus of control). In total, 14 pairs of pianists, who were not informed of the congruence manipulation, were placed back-to-back and were directed to achieve a successful performance over four repeated performances without verbal communication. Interpersonal coordination generally improved from the first to final performance in the congruent and incongruent conditions for both the tempo and dynamics tasks. Furthermore, in incongruent conditions, results suggest that performers prioritise the joint performance in the tempo task, but prioritise their own performance in the dynamics task. Although individual performance appears to be modulated by musical expertise, the balance of individual/joint performance across the duo is not influenced by musical part (melody vs accompaniment), expressive instruction, musical expertise or locus of control.

Investigation of Perceptual-Motor Behavior Across the Expert Athlete to Disabled Patient Skill Continuum can Advance Theory and Practical Application

A framework is presented of how theoretical predictions can be tested across the expert athlete to disabled patient skill continuum. Common-coding theory is used as the exemplar to discuss sensory and motor system contributions to perceptual-motor behavior. Behavioral and neural studies investigating expert athletes and patients recovering from cerebral stroke are reviewed. They provide evidence of bi-directional contributions of visual and motor systems to perceptual-motor behavior. Majority of this research is focused on perceptual-motor performance or learning, with less on transfer. The field is ripe for research designed to test theoretical predictions across the expert athlete to disabled patient skill continuum. Our view has implications for theory and practice in sports science, physical education, and rehabilitation.

Pointing out mechanisms underlying joint action

Recent studies have shown that spatial and feature-based attention can contribute to inducing a spatial compatibility effect in both the standard Simon task and the joint Simon task. Less work generally has been devoted to investigating how attention modulates spatial compatibility effects. In the present study, we aimed to explore whether indirectly manipulating the degree of attention necessary to respond to a compatible or an incompatible stimulus can modulate the spatial compatibility effect in a joint Simon task and elicit a compatibility effect in the individual go-no-go Simon task. To this end, we biased spatial attention to the compatible stimulus by asking participants to perform a pointing response always toward the compatible side, regardless of the stimulus location. Crucially, reaction times-recorded at gesture onset-showed a compatibility effect pattern in the individual condition and an additional modulation in the joint condition. These results show that the spatial attention intrinsic to action planning can affect both individual and joint Simon tasks.

No evidence of task co-representation in a joint Stroop task

People working together on a task must often represent the goals and salient items of their partner. The aim of the present study was to study the influence of joint task representations in an interference task in which the congruency relies on semantic identity. If task representations are shared between partners in a joint Stroop task (co-representation account), we hypothesized that items in the response set of one partner might influence performance of the other. In Experiment 1, pairs of participants sat side by side. Each participant was instructed to press one of two buttons to indicate which of two colors assigned to them was present, ignoring the text and responding only to the pixel color. There were three types of incongruent distractor words: names of colors from their own response set, names of colors from the other partner's response set, and neutral words for colors not used as font colors. The results of Experiment 1 showed that when people were doing this task together, distractor words from the partner's response set interfered more than neutral words and just as much as the words from their own response color set. However, in three follow-up experiments (Experiments 2a, 2b, and 2c), we found an elevated interference for the other response-set words even though no co-actor was present. The overall pattern of results across our study suggests that an alternative response set, regardless of whether it belonged to a co-actor or to a non-social no-go condition, evoked equal amounts of interference comparable to those of the own response set. Our findings are in line with a theory of common coding, in which all events-irrespective of their social nature-are represented and can influence behavior.

Social Transmission of Experience of Agency: An Experimental Study

The sense of controlling one’s own actions is fundamental to normal human mental function, and also underlies concepts of social responsibility for action. However, it remains unclear how the wider social context of human action influences sense of agency. Using a simple experimental design, we investigated, for the first time, how observing the action of another person or a robot could potentially influence one’s own sense of agency. We assessed how observing another’s action might change the perceived temporal relationship between one’s own voluntary actions and their outcomes, which has been proposed as an implicit measure of sense of agency. Working in pairs, participants chose between two action alternatives, one rewarded more frequently than the other, while watching a rotating clock hand. They judged, in separate blocks, either the time of their own action, or the time of a tone that followed the action. These were compared to baseline judgements of actions alone, or tones alone, to calculate the perceptual shift of action toward outcome and vice versa. Our design focused on how these two dependent variables, which jointly provide an implicit measure of sense of agency, might be influenced by observing another’s action. In the observational group, each participant could see the other’s actions. Multivariate analysis showed that the perceived time of action and tone shifted progressively toward the actual time of outcome with repeated experience of this social situation. No such progressive change occurred in other groups for whom a barrier hid participants’ actions from each other. However, a similar effect was observed in the group that viewed movements of a human-like robotic hand, rather than actions of another person. This finding suggests that observing the actions of others increases the salience of the external outcomes of action and this effect is not unique to observing human agents. Social contexts in which we see others controlling external events may play an important role in mentally representing the impact of our own actions on the external world.

Neuroimaging of the joint Simon effect with believed biological and non-biological co-actors

Performing a task alone or together with another agent can produce different outcomes. The current study used event-related functional magnetic resonance imaging (fMRI) to investigate the neural underpinnings when participants performed a Go/Nogo task alone or complementarily with another co-actor (unseen), whom was believed to be another human or a computer. During both complementary tasks, reaction time data suggested that participants integrated the potential action of their co-actor in their own action planning. Compared to the single-actor task, increased parietal and precentral activity during complementary tasks as shown in the fMRI data further suggested representation of the co-actor’s response. The superior frontal gyrus of the medial prefrontal cortex was differentially activated in the human co-actor condition compared to the computer co-actor condition. The medial prefrontal cortex, involved thinking about the beliefs and intentions of other people, possibly reflects a social-cognitive aspect or self-other discrimination during the joint task when believing a biological co-actor is present. Our results suggest that action co-representation can occur even offline with any agent type given a priori information that they are co-acting; however, additional regions are recruited when participants believe they are task-sharing with another human.