Predicting actions from subtle preparatory movements

Seeing social interactions

Seeing the interactions between other people is a critical part of our everyday visual experience, but recognizing the social interactions of others is often considered outside the scope of vision and grouped with higher-level social cognition like theory of mind. Recent work, however, has revealed that recognition of social interactions is efficient and automatic, is well modeled by bottom-up computational algorithms, and occurs in visually-selective regions of the brain. We review recent evidence from these three methodologies (behavioral, computational, and neural) that converge to suggest the core of social interaction perception is visual. We propose a computational framework for how this process is carried out in the brain and offer directions for future interdisciplinary investigations of social perception.

Behavioral optogenetics in nonhuman primates; a psychological perspective

Optogenetics has been a promising and developing technology in systems neuroscience throughout the past decade. It has been difficult though to reliably establish the potential behavioral effects of optogenetic perturbation of the neural activity in nonhuman primates. This poses a challenge on the future of optogenetics in humans as the concepts and technology need to be developed in nonhuman primates first. Here, I briefly summarize the viable approaches taken to improve nonhuman primate behavioral optogenetics, then focus on one approach: improvements in the measurement of behavior. I bring examples from visual behavior and show how the choice of method of measurement might conceal large behavioral effects. I will then discuss the "cortical perturbation detection" task in detail as an example of a sensitive task that can record the behavioral effects of optogenetic cortical stimulation with high fidelity. Finally, encouraged by the rich scientific landscape ahead of behavioral optogenetics, I invite technology developers to improve the chronically implantable devices designed for simultaneous neural recording and optogenetic intervention in nonhuman primates.

Human and macaque pairs employ different coordination strategies in a transparent decision game

Many real-world decisions in social contexts are made while observing a partner's actions. To study dynamic interactions during such decisions, we developed a setup where two agents seated face-to-face to engage in game-theoretical tasks on a shared transparent touchscreen display ('transparent games'). We compared human and macaque pairs in a transparent version of the coordination game 'Bach-or-Stravinsky', which entails a conflict about which of two individually-preferred opposing options to choose to achieve coordination. Most human pairs developed coordinated behavior and adopted dynamic turn-taking to equalize the payoffs. All macaque pairs converged on simpler, static coordination. Remarkably, two animals learned to coordinate dynamically after training with a human confederate. This pair selected the faster agent's preferred option, exhibiting turn-taking behavior that was captured by modeling the visibility of the partner's action before one's own movement. Such competitive turn-taking was unlike the prosocial turn-taking in humans, who equally often initiated switches to and from their preferred option. Thus, the dynamic coordination is not restricted to humans but can occur on the background of different social attitudes and cognitive capacities in rhesus monkeys. Overall, our results illustrate how action visibility promotes the emergence and maintenance of coordination when agents can observe and time their mutual actions.

Anticipation and compensation for somatosensory deficits in object handling: evidence from a patient with large fiber sensory neuropathy

The purpose of this study was to determine the contributions of feedforward and feedback processes on grip force regulation and object orientation during functional manipulation tasks. One patient with massive somatosensory loss resulting from large fiber sensory neuropathy and 10 control participants were recruited. Three experiments were conducted: 1) perturbation to static holding; 2) discrete vertical movement; and 3) functional grasp and place. The availability of visual feedback was also manipulated to assess the nature of compensatory mechanisms. Results from experiment 1 indicated that both the deafferented patient and controls used anticipatory grip force adjustments before self-induced perturbation to static holding. The patient exhibited increased grip response time, but the magnitude of grip force adjustments remained correlated with perturbation forces in the self-induced and external perturbation conditions. In experiment 2, the patient applied peak grip force substantially in advance of maximum load force. Unlike controls, the patient's ability to regulate object orientation was impaired without visual feedback. In experiment 3, the duration of unloading, transport, and release phases were longer for the patient, with increased deviation of object orientation at phase transitions. These findings show that the deafferented patient uses distinct modes of anticipatory control according to task constraints and that responses to perturbations are mediated by alternative afferent information. The loss of somatosensory feedback thus appears to impair control of object orientation, whereas variation in the temporal organization of functional tasks may reflect strategies to mitigate object instability associated with changes in movement dynamics.NEW & NOTEWORTHY This study evaluates the effects of sensory neuropathy on the scaling and timing of grip force adjustments across different object handling tasks (i.e., holding, vertical movement, grasping, and placement). In particular, these results illustrate how novel anticipatory and online control processes emerge to compensate for the loss of somatosensory feedback. In addition, we provide new evidence on the role of somatosensory feedback for regulating object orientation during functional prehensile movement.

Chimpanzees (Pan troglodytes) show subtle signs of uncertainty when choices are more difficult

Humans can tell when they find a task difficult. Subtle uncertainty behaviors like changes in motor speed and muscle tension precede and affect these experiences. Theories of animal metacognition likewise stress the importance of endogenous signals of uncertainty as cues that motivate metacognitive behaviors. However, while researchers have investigated second-order behaviors like information seeking and declining difficult trials in nonhuman animals, they have devoted little attention to the behaviors that express the cognitive conflict that gives rise to such behaviors in the first place. Here we explored whether three chimpanzees would, like humans, show hand wavering more when faced with more difficult choices in a touch screen transitive inference task. While accuracy was very high across all conditions, all chimpanzees wavered more frequently in trials that were objectively more difficult, demonstrating a signature behavior which accompanies experiences of difficulty in humans. This lends plausibility to the idea that feelings of uncertainty, like other emotions, can be studied in nonhuman animals. We propose to routinely assess uncertainty behaviors to inform models of procedural metacognition in nonhuman animals.

The ability to predict actions of others from distributed cues is still developing in 6- to 8-year-old children

Adults use distributed cues in the bodies of others to predict and counter their actions. To investigate the development of this ability, we had adults and 6- to 8-year-old children play a competitive game with a confederate who reached toward one of two targets. Child and adult participants, who sat across from the confederate, attempted to beat the confederate to the target by touching it before the confederate did. Adults used cues distributed through the head, shoulders, torso, and arms to predict the reaching actions. Children, in contrast, used cues in the arms and torso, but we did not find any evidence that they could use cues in the head or shoulders to predict the actions. These results provide evidence for a change in the ability to respond rapidly to predictive cues to others’ actions from childhood to adulthood. Despite humans’ sensitivity to action goals even in infancy, the ability to read cues from the body for action prediction in rapid interactive settings is still developing in children as old as 6 to 8 years of age.

Competition Rather Than Observation and Cooperation Facilitates Optimal Motor Planning

Humans tend to select motor planning with a high reward and low success compared with motor planning, which has a small reward and high success rate. Previous studies have shown such a risk-seeking property in motor decision tasks. However, it is unclear how to facilitate a shift from risk-seeking to optimal motor planning that maximizes the expected reward. Here, we investigate the effect of interacting with virtual partners/opponents on motor plans since interpersonal interaction has a powerful influence on human perception, action, and cognition. This study compared three types of interactions (competition, cooperation, and observation) and two types of virtual partners/opponents (those engaged in optimal motor planning and those engaged in risk-averse motor planning). As reported in previous studies, the participants took a risky aim point when they performed a motor decision task alone. However, we found that the participant's aim point was significantly modulated when they performed the same task while competing with a risk-averse opponent (p = 0.018) and that there was no significant difference from the optimal aim point (p = 0.63). No significant modulation in the aim points was observed during the cooperation and observation tasks. These results highlight the importance of competition for modulating suboptimal decision-making and optimizing motor performance.

Early warning signals in motion inference

The ability to infer intention lies at the basis of many social interactions played out via motor actions. We consider a simple paradigm of this ability in humans using data from experiments simulating an antagonistic game between an Attacker and a Blocker. Evidence shows early inference of an Attacker move by as much as 100ms but the nature of the informational cues signaling the impending move remains unknown. We show that the transition to action has the hallmark of a critical transition that is accompanied by early warning signals. These early warning signals occur as much as 130 ms before motion ensues-showing a sharp rise in motion autocorrelation at lag-1 and a sharp rise in the autocorrelation decay time. The early warning signals further correlate strongly with Blocker response times. We analyze the variance of the motion near the point of transition and find that it diverges in a manner consistent with the dynamics of a fold-transition. To test if humans can recognize and act upon these early warning signals, we simulate the dynamics of fold-transition events and ask people to recognize the onset of directional motion: participants react faster to fold-transition dynamics than to its uncorrelated counterpart. Together, our findings suggest that people can recognize the intent and onset of motion by inferring its early warning signals.

Subtle predictive movements reveal actions regardless of social context

Humans have a remarkable ability to predict the actions of others. To address what information enables this prediction and how the information is modulated by social context, we used videos collected during an interactive reaching game. Two participants (an “initiator” and a “responder”) sat on either side of a plexiglass screen on which two targets were affixed. The initiator was directed to tap one of the two targets, and the responder had to either beat the initiator to the target (competition) or arrive at the same time (cooperation). In a psychophysics experiment, new observers predicted the direction of the initiators' reach from brief clips, which were clipped relative to when the initiator began reaching. A machine learning classifier performed the same task. Both humans and the classifier were able to determine the direction of movement before the finger lift-off in both social conditions. Further, using an information mapping technique, the relevant information was found to be distributed throughout the body of the initiator in both social conditions. Our results indicate that we reveal our intentions during cooperation, in which communicating the future course of actions is beneficial, and also during competition despite the social motivation to reveal less information.

Optimizing motor decision-making through competition with opponents

Although optimal decision-making is essential for sports performance and fine motor control, it has been repeatedly confirmed that humans show a strong risk-seeking bias, selecting a risky strategy over an optimal solution. Despite such evidence, the ideal method to promote optimal decision-making remains unclear. Here, we propose that interactions with other people can influence motor decision-making and improve risk-seeking bias. We developed a competitive reaching game (a variant of the “chicken game”) in which aiming for greater rewards increased the risk of no reward and subjects competed for the total reward with their opponent. The game resembles situations in sports, such as a penalty kick in soccer, service in tennis, the strike zone in baseball, or take-off in ski jumping. In five different experiments, we demonstrated that, at the beginning of the competitive game, the subjects robustly switched their risk-seeking strategy to a risk-averse strategy. Following the reversal of the strategy, the subjects achieved optimal decision-making when competing with risk-averse opponents. This optimality was achieved by a non-linear influence of an opponent’s decisions on a subject’s decisions. These results suggest that interactions with others can alter human motor decision strategies and that competition with a risk-averse opponent is key for optimizing motor decision-making.

Emergence and suppression of cooperation by action visibility in transparent games

Real-world agents, humans as well as animals, observe each other during interactions and choose their own actions taking the partners' ongoing behaviour into account. Yet, classical game theory assumes that players act either strictly sequentially or strictly simultaneously without knowing each other's current choices. To account for action visibility and provide a more realistic model of interactions under time constraints, we introduce a new game-theoretic setting called transparent games, where each player has a certain probability of observing the partner's choice before deciding on its own action. By means of evolutionary simulations, we demonstrate that even a small probability of seeing the partner's choice before one's own decision substantially changes the evolutionary successful strategies. Action visibility enhances cooperation in an iterated coordination game, but reduces cooperation in a more competitive iterated Prisoner's Dilemma. In both games, "Win-stay, lose-shift" and "Tit-for-tat" strategies are predominant for moderate transparency, while a "Leader-Follower" strategy emerges for high transparency. Our results have implications for studies of human and animal social behaviour, especially for the analysis of dyadic and group interactions.

Decision-making in sensorimotor control

Skilled sensorimotor interactions with the world result from a series of decision-making processes that determine, on the basis of information extracted during the unfolding sequence of events, which movements to make and when and how to make them. Despite this inherent link between decision-making and sensorimotor control, research into each of these two areas has largely evolved in isolation, and it is only fairly recently that researchers have begun investigating how they interact and, together, influence behaviour. Here, we review recent behavioural, neurophysiological and computational research that highlights the role of decision-making processes in the selection, planning and control of goal-directed movements in humans and nonhuman primates.

Emergent Synergistic Grasp-Like Behavior in a Visuomotor Joint Action Task: Evidence for Internal Forward Models as Building Blocks of Human Interactions

Central to the mechanistic understanding of the human mind is to clarify how cognitive functions arise from simpler sensory and motor functions. A longstanding assumption is that forward models used by sensorimotor control to anticipate actions also serve to incorporate other people's actions and intentions, and give rise to sensorimotor interactions between people, and even abstract forms of interactions. That is, forward models could aid core aspects of human social cognition. To test whether forward models can be used to coordinate interactions, here we measured the movements of pairs of participants in a novel joint action task. For the task they collaborated to lift an object, each of them using fingers of one hand to push against the object from opposite sides, just like a single person would use two hands to grasp the object bimanually. Perturbations of the object were applied randomly as they are known to impact grasp-specific movement components in common grasping tasks. We found that co-actors quickly learned to make grasp-like movements with grasp components that showed coordination on average based on action observation of peak deviation and velocity of their partner's trajectories. Our data suggest that co-actors adopted pre-existing bimanual grasp programs for their own body to use forward models of their partner's effectors. This is consistent with the long-held assumption that human higher-order cognitive functions may take advantage of sensorimotor forward models to plan social behavior. New and Noteworthy: Taking an approach of sensorimotor neuroscience, our work provides evidence for a long-held belief that the coordination of physical as well as abstract interactions between people originates from certain sensorimotor control processes that form mental representations of people's bodies and actions, called forward models. With a new joint action paradigm and several new analysis approaches we show that, indeed, people coordinate each other's interactions based on forward models and mutual action observation.