Neural and Computational Mechanisms of Action Processing: Interaction between Visual and Motor Representations

Contextualizing predictive minds

The structure of human memory seems to be optimized for efficient prediction, planning, and behavior. We propose that these capacities rely on a tripartite structure of memory that includes concepts, events, and contexts-three layers that constitute the mental world model. We suggest that the mechanism that critically increases adaptivity and flexibility is the tendency to contextualize. This tendency promotes local, context-encoding abstractions, which focus event- and concept-based planning and inference processes on the task and situation at hand. As a result, cognitive contextualization offers a solution to the frame problem-the need to select relevant features of the environment from the rich stream of sensorimotor signals. We draw evidence for our proposal from developmental psychology and neuroscience. Adopting a computational stance, we present evidence from cognitive modeling research which suggests that context sensitivity is a feature that is critical for maximizing the efficiency of cognitive processes. Finally, we turn to recent deep-learning architectures which independently demonstrate how context-sensitive memory can emerge in a self-organized learning system constrained by cognitively-inspired inductive biases.

Kinematic coding: Measuring information in naturalistic behaviour

Recent years have seen an explosion of interest in naturalistic behaviour and in machine learning tools for automatically tracking it. However, questions about what to measure, how to measure it, and how to relate naturalistic behaviour to neural activity and cognitive processes remain unresolved. In this Perspective, we propose a general experimental and computational framework - kinematic coding - for measuring how information about cognitive states is encoded in structured patterns of behaviour and how this information is read out by others during social interactions. This framework enables the design of new experiments and the generation of testable hypotheses that link behaviour, cognition, and neural activity at the single-trial level. Researchers can employ this framework to identify single-subject, single-trial encoding and readout computations and address meaningful questions about how information encoded in bodily motion is transmitted and communicated.

Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning

Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.

From neural noise to co-adaptability: Rethinking the multifaceted architecture of motor variability

In the last decade, the source and the functional meaning of motor variability have attracted considerable attention in behavioral and brain sciences. This construct classically combined different levels of description, variable internal robustness or coherence, and multifaceted operational meanings. We provide here a comprehensive review of the literature with the primary aim of building a precise lexicon that goes beyond the generic and monolithic use of motor variability. In the pars destruens of the work, we model three domains of motor variability related to peculiar computational elements that influence fluctuations in motor outputs. Each domain is in turn characterized by multiple sub-domains. We begin with the domains of noise and differentiation. However, the main contribution of our model concerns the domain of adaptability, which refers to variation within the same exact motor representation. In particular, we use the terms learning and (social)fitting to specify the portions of motor variability that depend on our propensity to learn and on our largely constitutive propensity to be influenced by external factors. A particular focus is on motor variability in the context of the sub-domain named co-adaptability. Further groundbreaking challenges arise in the modeling of motor variability. Therefore, in a separate pars construens, we attempt to characterize these challenges, addressing both theoretical and experimental aspects as well as potential clinical implications for neurorehabilitation. All in all, our work suggests that motor variability is neither simply detrimental nor beneficial, and that studying its fluctuations can provide meaningful insights for future research.

Effector-specific motor simulation supplements core action recognition processes in adverse conditions

Observing other people acting activates imitative motor plans in the observer. Whether, and if so when and how, such 'effector-specific motor simulation' contributes to action recognition remains unclear. We report that individuals born without upper limbs (IDs)-who cannot covertly imitate upper-limb movements-are significantly less accurate at recognizing degraded (but not intact) upper-limb than lower-limb actions (i.e. point-light animations). This finding emphasizes the need to reframe the current controversy regarding the role of effector-specific motor simulation in action recognition: instead of focusing on the dichotomy between motor and non-motor theories, the field would benefit from new hypotheses specifying when and how effector-specific motor simulation may supplement core action recognition processes to accommodate the full variety of action stimuli that humans can recognize.

Tool mastering today - an interdisciplinary perspective

Tools have coined human life, living conditions, and culture. Recognizing the cognitive architecture underlying tool use would allow us to comprehend its evolution, development, and physiological basis. However, the cognitive underpinnings of tool mastering remain little understood in spite of long-time research in neuroscientific, psychological, behavioral and technological fields. Moreover, the recent transition of tool use to the digital domain poses new challenges for explaining the underlying processes. In this interdisciplinary review, we propose three building blocks of tool mastering: (A) perceptual and motor abilities integrate to tool manipulation knowledge, (B) perceptual and cognitive abilities to functional tool knowledge, and (C) motor and cognitive abilities to means-end knowledge about tool use. This framework allows for integrating and structuring research findings and theoretical assumptions regarding the functional architecture of tool mastering via behavior in humans and non-human primates, brain networks, as well as computational and robotic models. An interdisciplinary perspective also helps to identify open questions and to inspire innovative research approaches. The framework can be applied to studies on the transition from classical to modern, non-mechanical tools and from analogue to digital user-tool interactions in virtual reality, which come with increased functional opacity and sensorimotor decoupling between tool user, tool, and target. By working towards an integrative theory on the cognitive architecture of the use of tools and technological assistants, this review aims at stimulating future interdisciplinary research avenues.

Kinematic priming of action predictions

The ability to anticipate what others will do next is crucial for navigating social, interactive environments. Here, we develop an experimental and analytical framework to measure the implicit readout of prospective intention information from movement kinematics. Using a primed action categorization task, we first demonstrate implicit access to intention information by establishing a novel form of priming, which we term kinematic priming: subtle differences in movement kinematics prime action prediction. Next, using data collected from the same participants in a forced-choice intention discrimination task 1 h later, we quantify single-trial intention readout-the amount of intention information read by individual perceivers in individual kinematic primes-and assess whether it can be used to predict the amount of kinematic priming. We demonstrate that the amount of kinematic priming, as indexed by both response times (RTs) and initial fixations to a given probe, is directly proportional to the amount of intention information read by the individual perceiver at the single-trial level. These results demonstrate that human perceivers have rapid, implicit access to intention information encoded in movement kinematics and highlight the potential of our approach to reveal the computations that permit the readout of this information with single-subject, single-trial resolution.

Ultra-high field fMRI identifies an action-observation network in the common marmoset

The observation of others' actions activates a network of temporal, parietal and premotor/prefrontal areas in macaque monkeys and humans. This action-observation network (AON) has been shown to play important roles in social action monitoring, learning by imitation, and social cognition in both species. It is unclear whether a similar network exists in New-World primates, which separated from Old-Word primates ~35 million years ago. Here we used ultra-high field fMRI at 9.4 T in awake common marmosets (Callithrix jacchus) while they watched videos depicting goal-directed (grasping food) or non-goal-directed actions. The observation of goal-directed actions activates a temporo-parieto-frontal network, including areas 6 and 45 in premotor/prefrontal cortices, areas PGa-IPa, FST and TE in occipito-temporal region and areas V6A, MIP, LIP and PG in the occipito-parietal cortex. These results show overlap with the humans and macaques' AON, demonstrating the existence of an evolutionarily conserved network that likely predates the separation of Old and New-World primates.

Context expectation influences the gait pattern biomechanics

Beyond classical aspects related to locomotion (biomechanics), it has been hypothesized that walking pattern is influenced by a combination of distinct computations including online sensory/perceptual sampling and the processing of expectations (neuromechanics). Here, we aimed to explore the potential impact of contrasting scenarios ("risky and potentially dangerous" scenario; "safe and comfortable" scenario) on walking pattern in a group of healthy young adults. Firstly, and consistently with previous literature, we confirmed that the scenario influences gait pattern when it is recalled concurrently to participants' walking activity (motor interference). More intriguingly, our main result showed that participants' gait pattern is also influenced by the contextual scenario when it is evoked only before the start of walking activity (motor expectation). This condition was designed to test the impact of expectations (risky scenario vs. safe scenario) on gait pattern, and the stimulation that preceded walking activity served as prior. Noteworthy, we combined statistical and machine learning (Support-Vector Machine classifier) approaches to stratify distinct levels of analyses that explored the multi-facets architecture of walking. In a nutshell, our combined statistical and machine learning analyses converge in suggesting that walking before steps is not just a paradox.

A modality-independent proto-organization of human multisensory areas

The processing of multisensory information is based upon the capacity of brain regions, such as the superior temporal cortex, to combine information across modalities. However, it is still unclear whether the representation of coherent auditory and visual events requires any prior audiovisual experience to develop and function. Here we measured brain synchronization during the presentation of an audiovisual, audio-only or video-only version of the same narrative in distinct groups of sensory-deprived (congenitally blind and deaf) and typically developed individuals. Intersubject correlation analysis revealed that the superior temporal cortex was synchronized across auditory and visual conditions, even in sensory-deprived individuals who lack any audiovisual experience. This synchronization was primarily mediated by low-level perceptual features, and relied on a similar modality-independent topographical organization of slow temporal dynamics. The human superior temporal cortex is naturally endowed with a functional scaffolding to yield a common representation across multisensory events.

The acute effects of action observation training on upper extremity functions, cognitive processes and reaction times: a randomized controlled trial

Aim: To investigate the acute effects of action observation training on upper extremity functions, cognitive functions and response time in healthy, young adults. Materials & methods: A total of 60 participants were randomly divided into five groups: the self-action observation group, action observation group, action practice group, non-action observation group and control group. The Jebsen-Taylor hand function test (JTHFT), nine-hole peg test, serial reaction time task and d2 test of attention were applied to the participants before and after the interventions. Results: JTHFT performance with both non-dominant and dominant hands improved significantly compared with baseline in all groups (p < 0.001). JTHFT performance with non-dominant and dominant hands differed between the groups (p < 0.001). Conclusion: Action observation training seems to enhance the performance of upper extremity-related functions. Observing self-actions resulted in statistically significant positive changes in more variables compared with other methods. However, its clinical effectiveness over the other methods should be investigated in future long-term studies. Clinical Trial Registration: NCT04932057 (ClinicalTrials.gov).

Effective Connectivity of the Human Mirror Neuron System During Social Cognition

The human mirror neuron system (MNS) can be considered the neural basis of social cognition. Identifying the global network structure of this system can provide significant progress in the field. In this study, we use dynamic causal modeling (DCM) to determine the effective connectivity between central regions of the MNS for the first time during different social cognition tasks. Sixty-seven healthy participants completed fMRI scanning while performing social cognition tasks, including imitation, empathy and theory of mind. Superior temporal sulcus (STS), inferior parietal lobule (IPL) and Brodmann area 44 (BA44) formed the regions of interest for DCM. Varying connectivity patterns, 540 models were built and fitted for each participant. By applying group-level analysis, Bayesian model selection and Bayesian model averaging, the optimal family and model for all experimental tasks were found. For all social-cognitive processes, effective connectivity from STS to IPL and from STS to BA44 was found. For imitation, additional mutual connections occurred between STS and BA44, as well as BA44 and IPL. The results suggest inverse models in which the motor regions BA44 and IPL receive sensory information from the STS. In contrast, for imitation, a sensory loop with an exchange of motor-to-sensory and sensory-to-motor information seems to exist.

No evidence for somatosensory attenuation during action observation of self-touch

The discovery of mirror neurons in the macaque brain in the 1990s triggered investigations on putative human mirror neurons and their potential functionality. The leading proposed function has been action understanding: Accordingly, we understand the actions of others by 'simulating' them in our own motor system through a direct matching of the visual information to our own motor programmes. Furthermore, it has been proposed that this simulation involves the prediction of the sensory consequences of the observed action, similar to the prediction of the sensory consequences of our executed actions. Here, we tested this proposal by quantifying somatosensory attenuation behaviourally during action observation. Somatosensory attenuation manifests during voluntary action and refers to the perception of self-generated touches as less intense than identical externally generated touches because the self-generated touches are predicted from the motor command. Therefore, we reasoned that if an observer simulates the observed action and, thus, he/she predicts its somatosensory consequences, then he/she should attenuate tactile stimuli simultaneously delivered to his/her corresponding body part. In three separate experiments, we found a systematic attenuation of touches during executed self-touch actions, but we found no evidence for attenuation when such actions were observed. Failure to observe somatosensory attenuation during observation of self-touch is not compatible with the hypothesis that the putative human mirror neuron system automatically predicts the sensory consequences of the observed action. In contrast, our findings emphasize a sharp distinction between the motor representations of self and others.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Predicting other people’s upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person’s identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action’s duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects’ touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions’ goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

Prefrontal-amygdala circuits in social decision-making

An increasing amount of research effort is being directed toward investigating the neural bases of social cognition from a systems neuroscience perspective. Evidence from multiple animal species is beginning to provide a mechanistic understanding of the substrates of social behaviors at multiple levels of neurobiology, ranging from those underlying high-level social constructs in humans and their more rudimentary underpinnings in monkeys to circuit-level and cell-type-specific instantiations of social behaviors in rodents. Here we review literature examining the neural mechanisms of social decision-making in humans, non-human primates and rodents, focusing on the amygdala and the medial and orbital prefrontal cortical regions and their functional interactions. We also discuss how the neuropeptide oxytocin impacts these circuits and their downstream effects on social behaviors. Overall, we conclude that regulated interactions of neuronal activity in the prefrontal-amygdala pathways critically contribute to social decision-making in the brains of primates and rodents.

The Functional Tactile Object Recognition Test: A Unidimensional Measure With Excellent Internal Consistency for Haptic Sensing of Real Objects After Stroke

Introduction

Our hands, with their exquisite sensors, work in concert with our sensing brain to extract sensory attributes of objects as we engage in daily activities. One in two people with stroke experience impaired body sensation, with negative impact on hand use and return to previous valued activities. Valid, quantitative tools are critical to measure somatosensory impairment after stroke. The functional Tactile Object Recognition Test (fTORT) is a quantitative measure of tactile (haptic) object recognition designed to test one’s ability to recognize everyday objects across seven sensory attributes using 14 object sets. However, to date, knowledge of the nature of object recognition errors is limited, and the internal consistency of performance across item scores and dimensionality of the measure have not been established.

Objectives

To describe the original development and construction of the test, characterize the distribution and nature of performance errors after stroke, and to evaluate the internal consistency of item scores and dimensionality of the fTORT.

Method

Data from existing cohorts of stroke survivors (n = 115) who were assessed on the fTORT quantitative measure of sensory performance were extracted and pooled. Item and scale analyses were conducted on the raw item data. The distribution and type of errors were characterized.

Results

The 14 item sets of the fTORT form a well-behaved unidimensional scale and demonstrate excellent internal consistency (Cronbach alpha of 0.93). Deletion of any item failed to improve the Cronbach score. Most items displayed a bimodal score distribution, with function and attribute errors (score 0) or correct response (score 3) being most common. A smaller proportion of one- or two-attribute errors occurred. The total score range differentiated performance over a wide range of object recognition impairment.

Conclusion

Unidimensional scale and similar factor loadings across all items support simple addition of the 14 item scores on the fTORT. Therapists can use the fTORT to quantify impaired tactile object recognition in people with stroke based on the current set of items. New insights on the nature of haptic object recognition impairment after stroke are revealed.

Transient Disruption of the Inferior Parietal Lobule Impairs the Ability to Attribute Intention to Action

Although it is well established that fronto-parietal regions are active during action observation, whether they play a causal role in the ability to infer others' intentions from visual kinematics remains undetermined. In the experiments reported here, we combined offline continuous theta burst stimulation (cTBS) with computational modeling to reveal and causally probe single-trial computations in the inferior parietal lobule (IPL) and inferior frontal gyrus (IFG). Participants received cTBS over the left anterior IPL and the left IFG pars orbitalis in separate sessions before completing an intention discrimination task (discriminate intention of observed reach-to-grasp acts) or a kinematic discrimination task unrelated to intention (discriminate peak wrist height of the same acts). We targeted intention-sensitive regions whose fMRI activity, recorded when observing the same reach-to-grasp acts, could accurately discriminate intention. We found that transient disruption of activity of the left IPL, but not the IFG, impaired the observer's ability to attribute intention to action. Kinematic discrimination unrelated to intention, in contrast, was largely unaffected. Computational analyses of how encoding (mapping of intention to movement kinematics) and readout (mapping of kinematics to intention choices) intersect at the single-trial level revealed that IPL cTBS did not diminish the overall sensitivity of intention readout to movement kinematics. Rather, it selectively misaligned intention readout with respect to encoding, deteriorating mapping from informative kinematic features to intention choices. These results provide causal evidence of how the left anterior IPL computes mapping from kinematics to intentions.

Motor resonance in monkey parietal and premotor cortex during action observation: Influence of viewing perspective and effector identity

Observing others performing motor acts like grasping has been shown to elicit neural responses in the observer`s parieto-frontal motor network, which typically becomes active when the observer would perform these actions him/herself. While some human studies suggested strongest motor resonance during observation of first person or egocentric perspectives compared to third person or allocentric perspectives, other research either report the opposite or did not find any viewpoint-related preferences in parieto-premotor cortices. Furthermore, it has been suggested that these motor resonance effects are lateralized in the parietal cortex depending on the viewpoint and identity of the observed effector (left vs right hand). Other studies, however, do not find such straightforward hand identity dependent motor resonance effects. In addition to these conflicting findings in human studies, to date, little is known about the modulatory role of viewing perspective and effector identity (left or right hand) on motor resonance effects in monkey parieto-premotor cortices. Here, we investigated the extent to which different viewpoints of observed conspecific hand actions yield motor resonance in rhesus monkeys using fMRI. Observing first person, lateral and third person viewpoints of conspecific hand actions yielded significant activations throughout the so-called action observation network, including STS, parietal and frontal cortices. Although region-of-interest analysis of parietal and premotor motor/mirror neuron regions AIP, PFG and F5, showed robust responses in these regions during action observation in general, a clear preference for egocentric or allocentric perspectives was not evident. Moreover, except for lateralized effects due to visual field biases, motor resonance in the monkey brain during grasping observation did not reflect hand identity dependent coding.

Computation-Based Feature Representation of Body Expressions in the Human Brain

Humans and other primate species are experts at recognizing body expressions. To understand the underlying perceptual mechanisms, we computed postural and kinematic features from affective whole-body movement videos and related them to brain processes. Using representational similarity and multivoxel pattern analyses, we showed systematic relations between computation-based body features and brain activity. Our results revealed that postural rather than kinematic features reflect the affective category of the body movements. The feature limb contraction showed a central contribution in fearful body expression perception, differentially represented in action observation, motor preparation, and affect coding regions, including the amygdala. The posterior superior temporal sulcus differentiated fearful from other affective categories using limb contraction rather than kinematics. The extrastriate body area and fusiform body area also showed greater tuning to postural features. The discovery of midlevel body feature encoding in the brain moves affective neuroscience beyond research on high-level emotion representations and provides insights in the perceptual features that possibly drive automatic emotion perception.

Incomplete information about the partner affects the development of collaborative strategies in joint action

Physical interaction with a partner plays an essential role in our life experience and is the basis of many daily activities. When two physically coupled humans have different and partly conflicting goals, they face the challenge of negotiating some type of collaboration. This requires that both participants understand their partner’s state and current actions. But, how would the collaboration be affected if information about their partner were unreliable or incomplete? We designed an experiment in which two players (a dyad) are mechanically connected through a virtual spring, but cannot see each other. They were instructed to perform reaching movements with the same start and end position, but through different via-points. In different groups of dyads we varied the amount of information provided to each player about his/her partner: haptic only (the interaction force perceived through the virtual spring), visuo-haptic (the interaction force is also displayed on the screen), and partner visible (in addition to interaction force, partner position is continuously displayed on the screen). We found that incomplete information about the partner affects not only the speed at which collaboration is achieved (less information, slower learning), but also the actual collaboration strategy. In particular, incomplete or unreliable information leads to an interaction strategy characterized by alternating leader-follower roles. Conversely, more reliable information leads to more synchronous behaviors, in which no specific roles can be identified. Simulations based on a combination of game theory and Bayesian estimation suggested that synchronous behaviors correspond to optimal interaction (Nash equilibrium). Roles emerge as sub-optimal forms of interaction, which minimize the need to account for the partner. These findings suggest that collaborative strategies in joint action are shaped by the trade-off between the task requirements and the uncertainty of the information available about the partner.

Many activities in daily life involve physical interaction with a partner or opponent. In many situations, they have conflicting goals and need to negotiate some form of collaboration. Although very common, these situations have rarely been studied empirically. In this study, we specifically address what is a ‘optimal’ collaboration and how it can be achieved. We also address how developing a collaboration is affected by uncertainty about partner actions. Through a combination of empirical studies and computer simulations based on game theory, we show that subject pairs (dyads) are capable of developing stable collaborations, but the learned collaboration strategy depends on the reliability of the information about the partner. High-information dyads converge to optimal strategies in a game-theoretic sense. Low-information dyads converge to strategies that minimize the need to know about the partner. These findings are consistent with a game-theoretic learning model which relies on estimates of partner actions, but not partner goals. This similarity sheds some light on the minimal computational machinery which is necessary to an intelligent agent in order to develop stable physical collaborations with a human partner.

Local Dot Motion, Not Global Configuration, Determines Dogs' Preference for Point-Light Displays

Visual perception remains an understudied area of dog cognition, particularly the perception of biological motion where the small amount of previous research has created an unclear impression regarding dogs' visual preference towards different types of point-light displays. To date, no thorough investigation has been conducted regarding which aspects of the motion contained in point-light displays attract dogs. To test this, pet dogs (N = 48) were presented with pairs of point-light displays with systematic manipulation of motion features (i.e., upright or inverted orientation, coherent or scrambled configuration, human or dog species). Results revealed a significant effect of inversion, with dogs directing significantly longer looking time towards upright than inverted dog point-light displays; no effect was found for scrambling or the scrambling-inversion interaction. No looking time bias was found when dogs were presented with human point-light displays, regardless of their orientation or configuration. The results of the current study imply that dogs' visual preference is driven by the motion of individual dots in accordance with gravity, rather than the point-light display's global arrangement, regardless their long exposure to human motion.

Ultra-rapid object categorization in real-world scenes with top-down manipulations

Humans are able to achieve visual object recognition rapidly and effortlessly. Object categorization is commonly believed to be achieved by interaction between bottom-up and top-down cognitive processing. In the ultra-rapid categorization scenario where the stimuli appear briefly and response time is limited, it is assumed that a first sweep of feedforward information is sufficient to discriminate whether or not an object is present in a scene. However, whether and how feedback/top-down processing is involved in such a brief duration remains an open question. To this end, here, we would like to examine how different top-down manipulations, such as category level, category type and real-world size, interact in ultra-rapid categorization. We have constructed a dataset comprising real-world scene images with a built-in measurement of target object display size. Based on this set of images, we have measured ultra-rapid object categorization performance by human subjects. Standard feedforward computational models representing scene features and a state-of-the-art object detection model were employed for auxiliary investigation. The results showed the influences from 1) animacy (animal, vehicle, food), 2) level of abstraction (people, sport), and 3) real-world size (four target size levels) on ultra-rapid categorization processes. This had an impact to support the involvement of top-down processing when rapidly categorizing certain objects, such as sport at a fine grained level. Our work on human vs. model comparisons also shed light on possible collaboration and integration of the two that may be of interest to both experimental and computational vision researches. All the collected images and behavioral data as well as code and models are publicly available at https://osf.io/mqwjz/.

Stability and flexibility in multisensory sampling: insights from perceptual illusions

Neural, oscillatory, and computational counterparts of multisensory processing remain a crucial challenge for neuroscientists. Converging evidence underlines a certain efficiency in balancing stability and flexibility of sensory sampling, supporting the general idea that multiple parallel and hierarchically organized processing stages in the brain contribute to our understanding of the (sensory/perceptual) world. Intriguingly, how temporal dynamics impact and modulate multisensory processes in our brain can be investigated benefiting from studies on perceptual illusions.

Agent-based representations of objects and actions in the monkey pre-supplementary motor area

Information about objects around us is essential for planning actions and for predicting those of others. Here, we studied pre-supplementary motor area F6 neurons with a task in which monkeys viewed and grasped (or refrained from grasping) objects, and then observed a human doing the same task. We found "action-related neurons" encoding selectively monkey's own action [self-type (ST)], another agent's action [other-type (OT)], or both [self- and other-type (SOT)]. Interestingly, we found "object-related neurons" exhibiting the same type of selectivity before action onset: Indeed, distinct sets of neurons discharged when visually presented objects were targeted by the monkey's own action (ST), another agent's action (OT), or both (SOT). Notably, object-related neurons appear to signal self and other's intention to grasp and the most likely grip type that will be performed, whereas action-related neurons encode a general goal attainment signal devoid of any specificity for the observed grip type. Time-resolved cross-modal population decoding revealed that F6 neurons first integrate information about object and context to generate an agent-shared signal specifying whether and how the object will be grasped, which progressively turns into a broader agent-based goal attainment signal during action unfolding. Importantly, shared representation of objects critically depends upon their location in the observer's peripersonal space, suggesting an "object-mirroring" mechanism through which observers could accurately predict others' impending action by recruiting the same motor representation they would activate if they were to act upon the same object in the same context.

Observing Action Sequences Elicits Sequence-Specific Neural Representations in Frontoparietal Brain Regions

Learning new skills by watching others is important for social and motor development throughout the lifespan. Prior research has suggested that observational learning shares common substrates with physical practice at both cognitive and brain levels. In addition, neuroimaging studies have used multivariate analysis techniques to understand neural representations in a variety of domains, including vision, audition, memory, and action, but few studies have investigated neural plasticity in representational space. Therefore, although movement sequences can be learned by observing other people's actions, a largely unanswered question in neuroscience is how experience shapes the representational space of neural systems. Here, across a sample of male and female participants, we combined pretraining and posttraining fMRI sessions with 6 d of observational practice to determine whether the observation of action sequences elicits sequence-specific representations in human frontoparietal brain regions and the extent to which these representations become more distinct with observational practice. Our results showed that observed action sequences are modeled by distinct patterns of activity in frontoparietal cortex and that such representations largely generalize to very similar, but untrained, sequences. These findings advance our understanding of what is modeled during observational learning (sequence-specific information), as well as how it is modeled (reorganization of frontoparietal cortex is similar to that previously shown following physical practice). Therefore, on a more fine-grained neural level than demonstrated previously, our findings reveal how the representational structure of frontoparietal cortex maps visual information onto motor circuits in order to enhance motor performance.

SIGNIFICANCE STATEMENT Learning by watching others is a cornerstone in the development of expertise and skilled behavior. However, it remains unclear how visual signals are mapped onto motor circuits for such learning to occur. Here, we show that observed action sequences are modeled by distinct patterns of activity in frontoparietal cortex and that such representations largely generalize to very similar, but untrained, sequences. These findings advance our understanding of what is modeled during observational learning (sequence-specific information), as well as how it is modeled (reorganization of frontoparietal cortex is similar to that previously shown following physical practice). More generally, these findings demonstrate how motor circuit involvement in the perception of action sequences shows high fidelity to prior work, which focused on physical performance of action sequences.

Computational Feature Analysis of Body Movements Reveals Hierarchical Brain Organization

Social species spend considerable time observing the body movements of others to understand their actions, predict their emotions, watch their games, or enjoy their dance movements. Given the important information obtained from body movements, we still know surprisingly little about the details of brain mechanisms underlying movement perception. In this fMRI study, we investigated the relations between movement features obtained from automated computational analyses of video clips and the corresponding brain activity. Our results show that low-level computational features map to specific brain areas related to early visual- and motion-sensitive regions, while mid-level computational features are related to dynamic aspects of posture encoded in occipital–temporal cortex, posterior superior temporal sulcus and superior parietal lobe. Furthermore, behavioral features obtained from subjective ratings correlated with activity in higher action observation regions. Our computational feature-based analysis suggests that the neural mechanism of movement encoding is organized in the brain not so much by semantic categories than by feature statistics of the body movements.

Action Observation Areas Represent Intentions From Subtle Kinematic Features

Mirror neurons have been proposed to underlie humans' ability to understand others' actions and intentions. Despite 2 decades of research, however, the exact computational and neuronal mechanisms implied in this ability remain unclear. In the current study, we investigated whether, in the absence of contextual cues, regions considered to be part of the human mirror neuron system represent intention from movement kinematics. A total of 21 participants observed reach-to-grasp movements, performed with either the intention to drink or to pour while undergoing functional magnetic resonance imaging. Multivoxel pattern analysis revealed successful decoding of intentions from distributed patterns of activity in a network of structures comprising the inferior parietal lobule, the superior parietal lobule, the inferior frontal gyrus, and the middle frontal gyrus. Consistent with the proposal that parietal regions play a key role in intention understanding, classifier weights were higher in the inferior parietal region. These results provide the first demonstration that putative mirror neuron regions represent subtle differences in movement kinematics to read the intention of an observed motor act.

Investigating common coding of observed and executed actions in the monkey brain using cross-modal multi-variate fMRI classification

Mirror neurons are generally described as a neural substrate hosting shared representations of actions, by simulating or 'mirroring' the actions of others onto the observer's own motor system. Since single neuron recordings are rarely feasible in humans, it has been argued that cross-modal multi-variate pattern analysis (MVPA) of non-invasive fMRI data is a suitable technique to investigate common coding of observed and executed actions, allowing researchers to infer the presence of mirror neurons in the human brain. In an effort to close the gap between monkey electrophysiology and human fMRI data with respect to the mirror neuron system, here we tested this proposal for the first time in the monkey. Rhesus monkeys either performed reach-and-grasp or reach-and-touch motor acts with their right hand in the dark or observed videos of human actors performing similar motor acts. Unimodal decoding showed that both executed or observed motor acts could be decoded from numerous brain regions. Specific portions of rostral parietal, premotor and motor cortices, previously shown to house mirror neurons, in addition to somatosensory regions, yielded significant asymmetric action-specific cross-modal decoding. These results validate the use of cross-modal multi-variate fMRI analyses to probe the representations of own and others' actions in the primate brain and support the proposed mapping of others' actions onto the observer's own motor cortices.

Mirror Neurons Modeled Through Spike-Timing-Dependent Plasticity are Affected by Channelopathies Associated with Autism Spectrum Disorder

Mirror neurons fire action potentials both when the agent performs a certain behavior and watches someone performing a similar action. Here, we present an original mirror neuron model based on the spike-timing-dependent plasticity (STDP) between two morpho-electrical models of neocortical pyramidal neurons. Both neurons fired spontaneously with basal firing rate that follows a Poisson distribution, and the STDP between them was modeled by the triplet algorithm. Our simulation results demonstrated that STDP is sufficient for the rise of mirror neuron function between the pairs of neocortical neurons. This is a proof of concept that pairs of neocortical neurons associating sensory inputs to motor outputs could operate like mirror neurons. In addition, we used the mirror neuron model to investigate whether channelopathies associated with autism spectrum disorder could impair the modeled mirror function. Our simulation results showed that impaired hyperpolarization-activated cationic currents (Ih) affected the mirror function between the pairs of neocortical neurons coupled by STDP.

Are We "Motorically" Wired to Others? High-Level Motor Computations and Their Role in Autism

High-level motor computations reflect abstract components far apart from the mere motor performance. Neural correlates of these computations have been explored both in nonhuman and human primates, supporting the idea that our brain recruits complex nodes for motor representations. Of note, these computations have exciting implications for social cognition, and they also entail important challenges in the context of autism. Here, we focus on these challenges benefiting from recent studies addressing motor interference, motor resonance, and high-level motor planning. In addition, we suggest new ideas about how one maps and shares the (motor) space with others. Taken together, these issues inspire intriguing and fascinating questions about the social tendency of our high-level motor computations, and this tendency may indicate that we are "motorically" wired to others. Thus, after furnishing preliminary insights on putative neural nodes involved in these computations, we focus on how the hypothesized social nature of high-level motor computations may be anomalous or limited in autism, and why this represents a critical challenge for the future.

Action Categorization in Rhesus Monkeys: discrimination of grasping from non-grasping manual motor acts

The ability to recognize others’ actions is an important aspect of social behavior. While neurophysiological and behavioral research in monkeys has offered a better understanding of how the primate brain processes this type of information, further insight with respect to the neural correlates of action recognition requires tasks that allow recording of brain activity or perturbing brain regions while monkeys simultaneously make behavioral judgements about certain aspects of observed actions. Here we investigated whether rhesus monkeys could actively discriminate videos showing grasping or non-grasping manual motor acts in a two-alternative categorization task. After monkeys became proficient in this task, we tested their ability to generalize to a number of untrained, novel videos depicting grasps or other manual motor acts. Monkeys generalized to a wide range of novel human or conspecific grasping and non-grasping motor acts. They failed, however, for videos showing unfamiliar actions such as a non-biological effector performing a grasp, or a human hand touching an object with the back of the hand. This study shows the feasibility of training monkeys to perform active judgements about certain aspects of observed actions, instrumental for causal investigations into the neural correlates of action recognition.

Spatial and viewpoint selectivity for others' observed actions in monkey ventral premotor mirror neurons

The spatial location and viewpoint of observed actions are closely linked in natural social settings. For example, actions observed from a subjective viewpoint necessarily occur within the observer’s peripersonal space. Neurophysiological studies have shown that mirror neurons (MNs) of the monkey ventral premotor area F5 can code the spatial location of live observed actions. Furthermore, F5 MN discharge can also be modulated by the viewpoint from which filmed actions are seen. Nonetheless, whether and to what extent MNs can integrate viewpoint and spatial location of live observed actions remains unknown. We addressed this issue by comparing the activity of 148 F5 MNs while macaque monkeys observed an experimenter grasping in three different combinations of viewpoint and spatial location, namely, lateral view in the (1) extrapersonal and (2) peripersonal space and (3) subjective view in the peripersonal space. We found that the majority of MNs were space-selective (60.8%): those selective for the peripersonal space exhibited a preference for the subjective viewpoint both at the single-neuron and population level, whereas space-unselective neurons were view invariant. These findings reveal the existence of a previously neglected link between spatial and viewpoint selectivity in MN activity during live-action observation.

Neural reuse of action perception circuits for language, concepts and communication

Neurocognitive and neurolinguistics theories make explicit statements relating specialized cognitive and linguistic processes to specific brain loci. These linking hypotheses are in need of neurobiological justification and explanation. Recent mathematical models of human language mechanisms constrained by fundamental neuroscience principles and established knowledge about comparative neuroanatomy offer explanations for where, when and how language is processed in the human brain. In these models, network structure and connectivity along with action- and perception-induced correlation of neuronal activity co-determine neurocognitive mechanisms. Language learning leads to the formation of action perception circuits (APCs) with specific distributions across cortical areas. Cognitive and linguistic processes such as speech production, comprehension, verbal working memory and prediction are modelled by activity dynamics in these APCs, and combinatorial and communicative-interactive knowledge is organized in the dynamics within, and connections between APCs. The network models and, in particular, the concept of distributionally-specific circuits, can account for some previously not well understood facts about the cortical 'hubs' for semantic processing and the motor system's role in language understanding and speech sound recognition. A review of experimental data evaluates predictions of the APC model and alternative theories, also providing detailed discussion of some seemingly contradictory findings. Throughout, recent disputes about the role of mirror neurons and grounded cognition in language and communication are assessed critically.

Action perception as hypothesis testing

We present a novel computational model that describes action perception as an active inferential process that combines motor prediction (the reuse of our own motor system to predict perceived movements) and hypothesis testing (the use of eye movements to disambiguate amongst hypotheses). The system uses a generative model of how (arm and hand) actions are performed to generate hypothesis-specific visual predictions, and directs saccades to the most informative places of the visual scene to test these predictions - and underlying hypotheses. We test the model using eye movement data from a human action observation study. In both the human study and our model, saccades are proactive whenever context affords accurate action prediction; but uncertainty induces a more reactive gaze strategy, via tracking the observed movements. Our model offers a novel perspective on action observation that highlights its active nature based on prediction dynamics and hypothesis testing.