Squeezing lemons in the bathroom: Contextual information modulates action recognition

Same Same, But Different: Brain Areas Underlying the Learning from Repetitive Episodic Prediction Errors

Prediction errors (PEs) function as learning signals. It is yet unclear how varying compared to repetitive PEs affect episodic memory in brain and behavior. The current study investigated cerebral and behavioral effects of experiencing either multiple alternative versions ("varying") or one single alternative version ("repetitive") of a previously encoded episode. Participants encoded a set of episodes ("originals") by watching videos showing toy stories. During scanning, participants either experienced originals, one single, or multiple alternative versions of the previously encoded episodes. Participants' memory performance was tested through recall of original objects. Varying and repetitive PEs revealed typical brain responses to the detection of mismatching information including inferior frontal and posterior parietal regions, as well as hippocampus, which is further linked to memory reactivation, and the amygdala, known for modulating memory consolidation. Furthermore, experiencing varying and repetitive PEs triggered distinct brain areas as revealed by direct contrast. Among others, experiencing varying versions triggered activity in the caudate, a region that has been associated with PEs. In contrast, repetitive PEs activated brain areas that resembled more those for retrieval of originally encoded episodes. Thus, ACC and posterior cingulate cortex activation seemed to serve both reactivating old and integrating new but similar information in episodic memory. Consistent with neural findings, participants recalled original objects less accurately when only presented with the same, but not varying, PE during fMRI. The current findings suggest that repeated PEs interact more strongly with a recalled original episodic memory than varying PEs.

Contextual expectations shape the motor coding of movement kinematics during the prediction of observed actions: A TMS study

Contextual information may shape motor resonance and support intention understanding during observation of incomplete, ambiguous actions. It is unclear, however, whether this effect is contingent upon kinematics ambiguity or contextual information is continuously integrated with kinematics to predict the overarching action intention. Moreover, a differentiation between the motor mapping of the intention suggested by context or kinematics has not been clearly demonstrated. In a first action execution phase, 29 participants were asked to perform reaching-to-grasp movements towards big or small food objects with the intention to eat or to move; electromyography from the First Dorsal Interosseous (FDI) and Abductor Digiti Minimi (ADM) was recorded. Depending on object size, the intentions to eat or to move were differently implemented by a whole-hand or a precision grip kinematics, thus qualifying an action-muscle dissociation. Then, in a following action prediction task, the same participants were asked to observe an actor performing the same actions and to predict his/her intention while motor resonance was assessed for the same muscles. Of note, videos were interrupted at early or late action phases, and actions were embedded in contexts pointing toward an eating or a moving intention, congruently or incongruently with kinematics. We found greater involvement of the FDI or ADM in the execution of precision or whole-hand grips, respectively. Crucially, this pattern of activation was mirrored during observation of the same actions in congruent contexts, but it was cancelled out or reversed in the incongruent ones, either when videos were interrupted at either early or long phases of action deployment. Our results extend previous evidence by showing that contextual information shapes motor resonance not only under conditions of perceptual uncertainty but also when more informative kinematics is available.

Differential contributions of body form, motion, and temporal information to subjective action understanding in naturalistic stimuli

Introduction

We investigated the factors underlying naturalistic action recognition and understanding, as well as the errors occurring during recognition failures.

Methods

Participants saw full-light stimuli of ten different whole-body actions presented in three different conditions: as normal videos, as videos with the temporal order of the frames scrambled, and as single static representative frames. After each stimulus presentation participants completed one of two tasks-a forced choice task where they were given the ten potential action labels as options, or a free description task, where they could describe the action performed in each stimulus in their own words.

Results

While generally, a combination of form, motion, and temporal information led to the highest action understanding, for some actions form information was sufficient and adding motion and temporal information did not increase recognition accuracy. We also analyzed errors in action recognition and found primarily two different types.

Discussion

One type of error was on the semantic level, while the other consisted of reverting to the kinematic level of body part processing without any attribution of semantics. We elaborate on these results in the context of naturalistic action perception.

Overlapping representations of observed actions and action-related features

The lateral occipitotemporal cortex (LOTC) has been shown to capture the representational structure of a smaller range of actions. In the current study, we carried out an fMRI experiment in which we presented human participants with images depicting 100 different actions and used representational similarity analysis (RSA) to determine which brain regions capture the semantic action space established using judgments of action similarity. Moreover, to determine the contribution of a wide range of action-related features to the neural representation of the semantic action space we constructed an action feature model on the basis of ratings of 44 different features. We found that the semantic action space model and the action feature model are best captured by overlapping activation patterns in bilateral LOTC and ventral occipitotemporal cortex (VOTC). An RSA on eight dimensions resulting from principal component analysis carried out on the action feature model revealed partly overlapping representations within bilateral LOTC, VOTC, and the parietal lobe. Our results suggest spatially overlapping representations of the semantic action space of a wide range of actions and the corresponding action-related features. Together, our results add to our understanding of the kind of representations along the LOTC that support action understanding.

Contextual Priors Shape Action Understanding before and beyond the Unfolding of Movement Kinematics

Previous studies have shown that contextual information may aid in guessing the intention underlying others' actions in conditions of perceptual ambiguity. Here, we aimed to evaluate the temporal deployment of contextual influence on action prediction with increasing availability of kinematic information during the observation of ongoing actions. We used action videos depicting an actor grasping an object placed on a container to perform individual or interpersonal actions featuring different kinematic profiles. Crucially, the container could be of different colors. First, in a familiarization phase, the probability of co-occurrence between each action kinematics and color cues was implicitly manipulated to 80% and 20%, thus generating contextual priors. Then, in a testing phase, participants were asked to predict action outcome when the same action videos were occluded at five different timeframes of the entire movement, ranging from when the actor was still to when the grasp of the object was fully accomplished. In this phase, all possible action-contextual cues' associations were equally presented. The results showed that for all occlusion intervals, action prediction was more facilitated when action kinematics deployed in high- than low-probability contextual scenarios. Importantly, contextual priors shaped action prediction even in the latest occlusion intervals, where the kinematic cues clearly unveiled an action outcome that was previously associated with low-probability scenarios. These residual contextual effects were stronger in individuals with higher subclinical autistic traits. Our findings highlight the relative contribution of kinematic and contextual information to action understanding and provide evidence in favor of their continuous integration during action observation.

The Representation of Observed Actions at the Subordinate, Basic, and Superordinate Level

Actions can be planned and recognized at different hierarchical levels, ranging from very specific (e.g., to swim backstroke) to very broad (e.g., locomotion). Understanding the corresponding neural representation is an important prerequisite to reveal how our brain flexibly assigns meaning to the world around us. To address this question, we conducted an event-related fMRI study in male and female human participants in which we examined distinct representations of observed actions at the subordinate, basic and superordinate level. Using multiple regression representational similarity analysis (RSA) in predefined regions of interest, we found that the three different taxonomic levels were best captured by patterns of activations in bilateral lateral occipitotemporal cortex (LOTC), showing the highest similarity with the basic level model. A whole-brain multiple regression RSA revealed that information unique to the basic level was captured by patterns of activation in dorsal and ventral portions of the LOTC and in parietal regions. By contrast, the unique information for the subordinate level was limited to bilateral occipitotemporal cortex, while no single cluster was obtained that captured unique information for the superordinate level. The behaviorally established action space was best captured by patterns of activation in the LOTC and superior parietal cortex, and the corresponding neural patterns of activation showed the highest similarity with patterns of activation corresponding to the basic level model. Together, our results suggest that occipitotemporal cortex shows a preference for the basic level model, with flexible access across the subordinate and the basic level.SIGNIFICANCE STATEMENT The human brain captures information at varying levels of abstraction. It is debated which brain regions host representations across different hierarchical levels, with some studies emphasizing parietal and premotor regions, while other studies highlight the role of the lateral occipitotemporal cortex (LOTC). To shed light on this debate, here we examined the representation of observed actions at the three taxonomic levels suggested by Rosch et al. (1976) Our results highlight the role of the LOTC, which hosts a shared representation across the subordinate and the basic level, with the highest similarity with the basic level model. These results shed new light on the hierarchical organization of observed actions and provide insights into the neural basis underlying the basic level advantage.

Finding Structure in Modern Dance

Research has shown that both adults and children organize familiar activity into discrete units with consistent boundaries, despite the dynamic, continuous nature of everyday experiences. However, less is known about how observers segment unfamiliar event sequences. In the current study, we took advantage of the novelty that is inherent in modern dance. Modern dance features natural human motion but does not contain canonical goals-therefore, observers cannot recruit prior goal-related knowledge to segment it. Our main aims were to identify whether observers segment modern dance into the steps intended by the dancers, and what types of cues contribute to segmentation under these circumstances. Experiment 1 used a classic event segmentation task and found that adults were able to consistently identify only a few of the dancers' intended steps. Experiment 2 tested adults in an offline labeling task. Results showed that steps which could more easily be labeled offline in Experiment 2 were more likely to be segmented online in Experiment 1.

The pain hidden in your hands: Facial expression of pain reduces the influence of goal-related information in action recognition

The involvement of the sensorimotor system in the perception of painful actions has been repeatedly demonstrated. Yet the cognitive processes corresponding to sensorimotor activations have not been identified. In particular, the respective role of higher-level and lower-level action representations such as goals and grips in the recognition of painful actions is not clear. Previous research has shown that in a neutral context, higher-level action representations (goals) are prioritized over lower-level action representations (grips) and guide action recognition. The present study evaluates to what extent the general priority given to goal-related information in the processing of visual actions can be modulated by a context of pain. We used the action violation paradigm developed by van Elk et al. (2008). In the present action tasks, participants had to judge whether the grip or the goal of object-directed actions displayed in photographs was correct or not. The actress in the photograph could show either a neutral facial expression or a facial expression of pain. In the control task, they had to judge whether the actress expressed pain. In the action tasks, goals influenced grip judgements more than grips influenced goal judgements overall, corroborating the priority given to goal-related information previously reported. Critically, the impact of irrelevant goal-related information on the identification of incorrect grips disappeared in the pain context. Moreover, judgements in the control task were similarly influenced by grip and goal-related information. Results suggest that a context of pain reduces the reliance on higher-level action for action judgments. Findings provide novel directions regarding the cognitive and brain mechanisms involved in action processing in painful situations and support pluralist views of action understanding.

Neural and behavioral signatures of the multidimensionality of manipulable object processing

Understanding how we recognize objects requires unravelling the variables that govern the way we think about objects and the neural organization of object representations. A tenable hypothesis is that the organization of object knowledge follows key object-related dimensions. Here, we explored, behaviorally and neurally, the multidimensionality of object processing. We focused on within-domain object information as a proxy for the decisions we typically engage in our daily lives - e.g., identifying a hammer in the context of other tools. We extracted object-related dimensions from subjective human judgments on a set of manipulable objects. We show that the extracted dimensions are cognitively interpretable and relevant - i.e., participants are able to consistently label them, and these dimensions can guide object categorization; and are important for the neural organization of knowledge - i.e., they predict neural signals elicited by manipulable objects. This shows that multidimensionality is a hallmark of the organization of manipulable object knowledge.

The cognitive structure underlying the organization of observed actions

In daily life, we frequently encounter actions performed by other people. Here we aimed to examine the key categories and features underlying the organization of a wide range of actions in three behavioral experiments (N = 378 participants). In Experiment 1, we used a multi-arrangement task of 100 different actions. Inverse multidimensional scaling and hierarchical clustering revealed 11 action categories, including Locomotion, Communication, and Aggressive actions. In Experiment 2, we used a feature-listing paradigm to obtain a wide range of action features that were subsequently reduced to 59 key features and used in a rating study (Experiment 3). A direct comparison of the feature ratings obtained in Experiment 3 between actions belonging to the categories identified in Experiment 1 revealed a number of features that appear to be critical for the distinction between these categories, e.g., the features Harm and Noise for the category Aggressive actions, and the features Targeting a person and Contact with others for the category Interaction. Finally, we found that a part of the category-based organization is explained by a combination of weighted features, whereas a significant proportion of variability remained unexplained, suggesting that there are additional sources of information that contribute to the categorization of observed actions. The characterization of action categories and their associated features serves as an important extension of previous studies examining the cognitive structure of actions. Moreover, our results may serve as the basis for future behavioral, neuroimaging and computational modeling studies.

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

Measuring the prediction of observed actions using an occlusion paradigm: Comparing autistic and non-autistic adults

Action prediction involves observing and predicting the actions of others and plays an important role in social cognition and interacting with others. It is thought to use simulation, whereby the observers use their own motor system to predict the observed actions. As individuals diagnosed with autism are characterized by difficulties understanding the actions of others and motor coordination issues, it is possible that action prediction ability is altered in this population. This study compared action prediction ability between 20 autistic and 22 non-autistic adults using an occlusion paradigm. Participants watched different videos of a female actor carrying out everyday actions. During each video, the action was transiently occluded by a gray rectangle for 1000 ms. During occlusions, the video was allowed to continue as normal or was moved forward (i.e., appearing to continue too far ahead) or moved backwards (i.e., appearing to continue too far behind). Participants were asked to indicate after each occlusion whether the action continued with the correct timing or was too far ahead/behind. Autistic individuals were less accurate than non-autistic individuals, particularly when the video was too far behind. A trend analysis suggested that autistic participants were more likely to judge too far behind occlusions as being in time. These preliminary results suggest that prediction ability may be altered in autistic adults, potentially due to slower simulation or a delayed onset of these processes. LAY SUMMARY: When we observe other people performing everyday actions, we use their movements to help us understand and predict what they are doing. In this study, we found that autistic compared to non-autistic adults were slightly less accurate at predicting other people's actions. These findings help to unpick the different ways that social understanding is affected in autism.

From Affordances to Abstract Words: The Flexibility of Sensorimotor Grounding

The sensorimotor system plays a critical role in several cognitive processes. Here, we review recent studies documenting this interplay at different levels. First, we concentrate on studies that have shown how the sensorimotor system is flexibly involved in interactions with objects. We report evidence demonstrating how social context and situations influence affordance activation, and then focus on tactile and kinesthetic components in body-object interactions. Then, we turn to word use, and review studies that have shown that not only concrete words, but also abstract words are grounded in the sensorimotor system. We report evidence that abstract concepts activate the mouth effector more than concrete concepts, and discuss this effect in light of studies on adults, children, and infants. Finally, we pinpoint possible sensorimotor mechanisms at play in the acquisition and use of abstract concepts. Overall, we show that the involvement of the sensorimotor system is flexibly modulated by context, and that its role can be integrated and flanked by that of other systems such as the linguistic system. We suggest that to unravel the role of the sensorimotor system in cognition, future research should fully explore the complexity of this intricate, and sometimes slippery, relation.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

What modulates the Mirror Neuron System during action observation?: Multiple factors involving the action, the actor, the observer, the relationship between actor and observer, and the context

Seeing an agent perform an action typically triggers a motor simulation of that action in the observer's Mirror Neuron System (MNS). Over the past few years, it has become increasingly clear that during action observation the patterns and strengths of responses in the MNS are modulated by multiple factors. The first aim of this paper is therefore to provide the most comprehensive survey to date of these factors. To that end, 22 distinct factors are described, broken down into the following sets: six involving the action; two involving the actor; nine involving the observer; four involving the relationship between actor and observer; and one involving the context. The second aim is to consider the implications of these findings for four prominent theoretical models of the MNS: the Direct Matching Model; the Predictive Coding Model; the Value-Driven Model; and the Associative Model. These assessments suggest that although each model is supported by a wide range of findings, each one is also challenged by other findings and relatively unaffected by still others. Hence, there is now a pressing need for a richer, more inclusive model that is better able to account for all of the modulatory factors that have been identified so far.

Adults, but not preschoolers or toddlers integrate situational constraints in their action anticipations: a developmental study on the flexibility of anticipatory gaze

Recent theories stress the role of situational information in understanding others' behaviour. For example, the predictive coding framework assumes that people take contextual information into account when anticipating other's actions. Likewise, the teleological stance theory assumes an early developing ability to consider situational constraints in action prediction. The current study investigates, over a wide age range, whether humans flexibly integrate situational constraints in their action anticipations. By means of an eye-tracking experiment, 2-year-olds, 5-year-olds, younger and older adults (together N = 181) observed an agent repeatedly taking one of two paths to reach a goal. Then, this path became blocked, and for test trials only the other path was passable. Results demonstrated that in test trials younger and older adults anticipated that the agent would take the continuous path, indicating that they took the situational constraints into account. In contrast, 2- and 5-year-olds anticipated that the agent would take the blocked path, indicating that they still relied on the agent's previous observed behaviour and-contrary to claims by the teleological stance theory-did not take the situational constraints into account. The results highlight developmental changes in human's ability to include situational constraints in their visual anticipations. Overall, the study contributes to theories on predictive coding and the development of action understanding.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Predicting other people’s upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person’s identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action’s duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects’ touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions’ goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

Neural dynamics of grip and goal integration during the processing of others' actions with objects: An ERP study

Recent behavioural evidence suggests that when processing others’ actions, motor acts and goal-related information both contribute to action recognition. Yet the neuronal mechanisms underlying the dynamic integration of the two action dimensions remain unclear. This study aims to elucidate the ERP components underlying the processing and integration of grip and goal-related information. The electrophysiological activity of 28 adults was recorded during the processing of object-directed action photographs (e.g., writing with pencil) containing either grip violations (e.g. upright pencil grasped with atypical-grip), goal violations (e.g., upside-down pencil grasped with typical-grip), both grip and goal violations (e.g., upside-down pencil grasped with atypical-grip), or no violations. Participants judged whether actions were overall typical or not according to object typical use. Brain activity was sensitive to the congruency between grip and goal information on the N400, reflecting the semantic integration between the two dimensions. On earlier components, brain activity was affected by grip and goal typicality independently. Critically, goal typicality but not grip typicality affected brain activity on the N300, supporting an earlier role of goal-related representations in action recognition. Findings provide new insights on the neural temporal dynamics of the integration of motor acts and goal-related information during the processing of others’ actions.

Predictive Impact of Contextual Objects during Action Observation: Evidence from Functional Magnetic Resonance Imaging

The processing of congruent stimuli, such as an object or action in its typical location, is usually associated with reduced neural activity, probably due to facilitated recognition. However, in some situations, congruency increases neural activity—for example, when objects next to observed actions are likely versus unlikely to be involved in forthcoming action steps. Here, we investigated using fMRI whether the processing of contextual cues during action perception is driven by their (in)congruency and, thus, informative value to make sense of an observed scene. Specifically, we tested whether both highly congruent contextual objects (COs), which strongly indicate a future action step, and highly incongruent COs, which require updating predictions about possible forthcoming action steps, provide more anticipatory information about the action course than moderately congruent COs. In line with our hypothesis that especially the inferior frontal gyrus (IFG) subserves the integration of the additional information into the predictive model of the action, we found highly congruent and incongruent COs to increase bilateral activity in action observation nodes, that is, the IFG, the occipitotemporal cortex, and the intraparietal sulcus. Intriguingly, BA 47 was significantly stronger engaged for incongruent COs reflecting the updating of prediction in response to conflicting information. Our findings imply that the IFG reflects the informative impact of COs on observed actions by using contextual information to supply and update the currently operating predictive model. In the case of an incongruent CO, this model has to be reconsidered and extended toward a new overarching action goal.

Contextual priors do not modulate action prediction in children with autism

Bayesian accounts of autism suggest that this disorder may be rooted in an impaired ability to estimate the probability of future events, possibly owing to reduced priors. Here, we tested this hypothesis within the action domain in children with and without autism using a behavioural paradigm comprising a familiarization and a testing phase. During familiarization, children observed videos depicting a child model performing actions in diverse contexts. Crucially, within this phase, we implicitly biased action-context associations in terms of their probability of co-occurrence. During testing, children observed the same videos but drastically shortened (i.e. reduced amount of kinematics information) and were asked to infer action unfolding. Since during the testing phase movement kinematics became ambiguous, we expected children's responses to be biased to contextual priors, thus compensating for perceptual uncertainty. While this probabilistic effect was present in controls, no such modulation was observed in autistic children, overall suggesting an impairment in using contextual priors when predicting other peoples' actions in uncertain environments.

Taking it out of context: The role of contextual coherence during social event segmentation

Social event segmentation, or parsing of the ongoing dynamic content into discrete social events, is thought to represent a mechanism that supports the expert human ability to navigate complex social environments. Here, we examined whether this ability is influenced by the temporal coherence of the context and by different sources of perceptual information. To do so, we created two video clips, one in which several situations unfolded in a contextually consistent manner, and the other in which the order of these situations was scrambled using a random sequence. Participants viewed each clip and were asked to mark social and nonsocial events in counterbalanced blocks of trials. We analyzed key-press behaviour as well as visual and auditory signals within the clips. Results showed that participants agreed on similar social and nonsocial events regardless of context availability, with greater agreement for social relative to nonsocial events. Context, however, modulated the reliance on sources of perceptual information, such that visual and auditory information was used differently when context was unavailable. Together, these data show that contextual coherence does not determine social event segmentation but serves a modulatory role in perceivers' reliance on perceptual sources of information when identifying events in complex social environments.

Uncertainty in contextual and kinematic cues jointly modulates motor resonance in primary motor cortex

Contextual information accompanying others' actions modulates "motor resonance", i.e., neural activity within motor areas that is elicited by movement observation. One possibility is that we weigh and combine such information in a Bayesian manner according to their relative uncertainty. Therefore, contextual information becomes particularly useful when others' actions are difficult to discriminate. It is unclear, however, whether this uncertainty modulates the neural activity in primary motor cortex (M1) during movement observation. Here, we applied single-pulse transcranial magnetic stimulation (TMS) while subjects watched different grasping actions. We operationalized motor resonance as grip-specific modulation of corticomotor excitability measured in the index (FDI) versus the little finger abductor (ADM). We experimentally modulated either the availability of kinematic information ( experiment 1) or the reliability of contextual cues ( experiment 2). Our results indicate that even in the absence of movement kinematics, reliable contextual information is enough to trigger significant muscle-specific corticomotor excitability changes in M1, which are strongest when both kinematics and contextual information are available. These findings suggest that bottom-up mechanisms that activate motor representations as a function of the observed kinematics and top-down mechanisms that activate motor representations associated with arbitrary cues converge in M1. NEW & NOTEWORTHY Our study reveals new neurophysiological insights in support of the Bayesian account of action observation by showing that "motor resonance", i.e., neural activity evoked by observing others' actions, incorporates the uncertainty related to both contextual (prior beliefs) and kinematic (sensory evidence) cues. Notably, we show that muscle-specific modulation of M1 is strongest when context and movement kinematics are available, and it can be elicited even in the absence of movement kinematics.

Observation of an action with a congruent contextual background facilitates corticospinal excitability: A combined TMS and eye-tracking experiment

Action observation produces activity in similar regions of the brain to those involved in action execution and can offer an effective intervention for motor (re)learning, although optimal viewing conditions for such interventions remain to be established. In this experiment, single-pulse transcranial magnetic stimulation (TMS) and eye-tracking were used simultaneously to investigate the effect of manipulating background context on both corticospinal excitability and visual attention during action observation. Twenty-four participants observed four different videos: (i) a static hand holding a sponge (control condition); and an index finger-thumb pinch of a sponge against (ii) a plain black background; (iii) a background containing objects that were incongruent with the observed action; and (iv) a background containing objects that were congruent with the observed action. TMS was delivered to the hand representation of the left primary motor cortex, and motor evoked potentials were recorded from the first dorsal interosseous and abductor digiti minimi muscles of the right hand. Eye movements were recorded throughout the experiment. Results indicated that corticospinal excitability was facilitated during the congruent context condition compared to both the static hand and plain black background conditions. In addition, the number of fixations and percentage of time participants spent fixating on the background scene were significantly greater during the incongruent and congruent conditions compared to the static hand and plain black background conditions. These results indicate that the provision of additional visual information that is congruent with the observed movement contributes to a facilitation of corticospinal excitability by providing the observer with information regarding the goal and intention of the observed action. Providing congruent contextual information may enhance the efficacy of action observation interventions for motor (re)learning.

Seeing for speaking: Semantic and lexical information provided by briefly presented, naturalistic action scenes

At the interface between scene perception and speech production, we investigated how rapidly action scenes can activate semantic and lexical information. Experiment 1 examined how complex action-scene primes, presented for 150 ms, 100 ms, or 50 ms and subsequently masked, influenced the speed with which immediately following action-picture targets are named. Prime and target actions were either identical, showed the same action with different actors and environments, or were unrelated. Relative to unrelated primes, identical and same-action primes facilitated naming the target action, even when presented for 50 ms. In Experiment 2, neutral primes assessed the direction of effects. Identical and same-action scenes induced facilitation but unrelated actions induced interference. In Experiment 3, written verbs were used as targets for naming, preceded by action primes. When target verbs denoted the prime action, clear facilitation was obtained. In contrast, interference was observed when target verbs were phonologically similar, but otherwise unrelated, to the names of prime actions. This is clear evidence for word-form activation by masked action scenes. Masked action pictures thus provide conceptual information that is detailed enough to facilitate apprehension and naming of immediately following scenes. Masked actions even activate their word-form information-as is evident when targets are words. We thus show how language production can be primed with briefly flashed masked action scenes, in answer to long-standing questions in scene processing.

Making sense of objects lying around: How contextual objects shape brain activity during action observation

Action recognition involves not only the readout of body movements and involved objects but also the integration of contextual information, e.g. the environment in which an action takes place. Notably, inferring superordinate goals and generating predictions about forthcoming action steps should benefit from screening the actor's immediate environment, in particular objects located in the actor's peripersonal space and thus potentially used in following action steps. Critically, if such contextual objects (COs) afford actions that are semantically related to the observed action, they may trigger or facilitate the inference of goals and the prediction of following actions. This fMRI study investigated the neural mechanisms underlying the integration of COs in semantic and spatial relation to observed actions. Specifically, we tested the hypothesis that the inferior frontal gyrus (IFG) subserves this integration. Participants observed action videos in which COs and observed actions had common overarching goals or not (goal affinity) and varied in their location relative to the actor. High goal affinity increased bilateral activity in action observation network nodes, i.e. the occipitotemporal cortex and the intraparietal sulcus, but also in the precuneus and middle frontal gyri. This finding suggests that the semantic relation between COs and actions is considered during action observation and triggers (rather than facilitates) processes beyond those usually involved in action observation. Moreover, COs with high goal affinity located close to the actor's dominant hand additionally engaged bilateral IFG, corroborating the view that IFG is critically involved in the integration of action steps under a common overarching goal.

Motor activation during action perception depends on action interpretation

Since the discovery of motor mirroring, the involvement of the motor system in action interpretation has been widely discussed. While some theories proposed that motor mirroring underlies human action understanding, others suggested that it is a corollary of action interpretation. We put these two accounts to the test by employing superficially similar actions that invite radically different interpretations of the underlying intentions. Using an action-observation task, we assessed motor activation (as indexed by the suppression of the EEG mu rhythm) in response to actions typically interpreted as instrumental (e.g., grasping) or referential (e.g., pointing) towards an object. Only the observation of instrumental actions resulted in enhanced mu suppression. In addition, the exposure to grasping actions failed to elicit mu suppression when they were preceded by speech, suggesting that the presence of communicative signals modulated the interpretation of the observed actions. These results suggest that the involvement of sensorimotor cortices during action processing is conditional on a particular (instrumental) action interpretation, and that action interpretation relies on inferential processes and top-down mechanisms that are implemented outside of the motor system.

Tracking the Time Course of Top-Down Contextual Effects on Motor Responses during Action Comprehension

Context plays a key role in coding high-level components of others' behavior, including the goal and the intention of an observed action. However, little is known about its possible role in shaping lower levels of action processing, such as simulating action kinematics and muscular activity. Furthermore, there is no evidence regarding the time course and the neural mechanisms subserving this modulation. To address these issues, we combined single-pulse transcranial magnetic stimulation and motor-evoked potentials while healthy humans watched videos of everyday actions embedded in congruent, incongruent, or ambiguous contexts. Video endings were occluded from view and participants had to predict action unfolding. Transcranial magnetic stimulation was delivered at 80, 240, and 400 ms after action onset. An earlier selective facilitation of motor resonance occurring at 240 ms was observed for actions embedded in congruent contexts, compared with those occurring in incongruent and ambiguous ones. Later on, at 400 ms, a selective inhibition of motor resonance was found for actions embedded in incongruent contexts, compared with those taking place in congruent and ambiguous ones. No modulations were observed at 80 ms. Together, these findings indicate that motor resonance can be modulated by contextual information with different timings, depending on the (in)congruency between the different levels of action representation. Furthermore, the different time course of these effects suggests that they stem from partially independent mechanisms, with the early facilitation directly involving M1, and the later inhibition recruiting high-level structures outside the motor system.

SIGNIFICANCE STATEMENT

Previous studies indicate that, when we observe other people's actions, the context in which actions take place influences intention understanding. However, little is known about the precise mechanisms involved in the contextual modulation of action representation (i.e., inhibition vs facilitation) and how they unfold in time. The present study sheds light on these aspects. Specifically, we show an early top-down facilitation (at ∼240 ms) and a later inhibition (at ∼400 ms) of motor resonance in response to actions observed in congruent and incongruent contexts, respectively.

Neural correlates of evoked phantom limb sensations

•

We present a neural network related to evoked phantom sensations in amputees.

•

Such networks were not related to the stimulation from the residual limb.

•

Difference in intra- and inter-hemispheric interactions between amputees and yoked controls.

•

This finding yields novel insights into the neural basis of phantom sensation.

Previous work showed the existence of changes in the topographic organization within the somatosensory cortex (SI) in amputees with phantom limb pain, however, the link between nonpainful phantom sensations such as cramping or tingling or the percept of the limb and cortical changes is less clear.

We used functional magnetic resonance imaging (fMRI) in a highly selective group of limb amputees who experienced inducible and reproducible nonpainful phantom sensations. A standardized procedure was used to locate body sites eliciting phantom sensations in each amputee. Selected body sites that could systematically evoke phantom sensations were stimulated using electrical pulses in order to induce phasic phantom sensations. Homologous body parts were also stimulated in a group of matched controls.

Activations related to evoked phantom sensations were found bilaterally in SI and the intraparietal sulci (IPS), which significantly correlated with the intensity of evoked phantom sensations. In addition, we found differences in intra- and interhemispheric interaction between amputees and controls during evoked phantom sensations. We assume that phantom sensations might be associated with a functional decoupling between bilateral SI and IPS, possibly resulting from transcallosal reorganization mechanisms following amputation.

Gaze Following Is Modulated by Expectations Regarding Others' Action Goals

Humans attend to social cues in order to understand and predict others’ behavior. Facial expressions and gaze direction provide valuable information to infer others’ mental states and intentions. The present study examined the mechanism of gaze following in the context of participants’ expectations about successive action steps of an observed actor. We embedded a gaze-cueing manipulation within an action scenario consisting of a sequence of naturalistic photographs. Gaze-induced orienting of attention (gaze following) was analyzed with respect to whether the gaze behavior of the observed actor was in line or not with the action-related expectations of participants (i.e., whether the actor gazed at an object that was congruent or incongruent with an overarching action goal). In Experiment 1, participants followed the gaze of the observed agent, though the gaze-cueing effect was larger when the actor looked at an action-congruent object relative to an incongruent object. Experiment 2 examined whether the pattern of effects observed in Experiment 1 was due to covert, rather than overt, attentional orienting, by requiring participants to maintain eye fixation throughout the sequence of critical photographs (corroborated by monitoring eye movements). The essential pattern of results of Experiment 1 was replicated, with the gaze-cueing effect being completely eliminated when the observed agent gazed at an action-incongruent object. Thus, our findings show that covert gaze following can be modulated by expectations that humans hold regarding successive steps of the action performed by an observed agent.

Objects Mediate Goal Integration in Ventrolateral Prefrontal Cortex during Action Observation

Actions performed by others are mostly not observed in isolation, but embedded in sequences of actions tied together by an overarching goal. Therefore, preceding actions can modulate the observer's expectations in relation to the currently perceived action. Ventrolateral prefrontal cortex (vlPFC), and inferior frontal gyrus (IFG) in particular, is suggested to subserve the integration of episodic as well as semantic information and memory, including action scripts. The present fMRI study investigated if activation in IFG varies with the effort to integrate expected and observed action, even when not required by the task. During an fMRI session, participants were instructed to attend to short videos of single actions and to deliver a judgment about the actor’s current goal. We manipulated the strength of goal expectation induced by the preceding action, implementing the parameter "goal-relatedness" between the preceding and the currently observed action. Moreover, since objects point to the probability of certain actions, we also manipulated whether the current and the preceding action shared at least one object or not. We found an interaction between the two factors goal-relatedness and shared object: IFG activation increased the weaker the goal-relatedness between the preceding and the current action was, but only when they shared at least one object. Here, integration of successive action steps was triggered by the re-appearing (shared) object but hampered by a weak goal-relatedness between the actually observed manipulation. These findings foster the recently emerging view that IFG is enhanced by goal-related conflicts during action observation.

Interacting hands: the role of attention for the joint Simon effect

Recent research in monkeys and humans has shown that the presence of the hands near an object enhances spatial processing for objects presented near the hand. This study aimed to test the effect of hand position on the joint Simon effect. In Experiment 1, two human co-actors shared a Simon task while placing their response hands either near the objects appearing on the monitor or away from the monitor. Experiment 2 varied each co-actor’s hand position independently. Experiment 3 tested whether enhanced spatial processing for objects presented near the hand is obtained when replacing one of the two co-actors by a non-human event-producing rubber hand. Experiment 1 provided evidence for a Simon effect. Hand position significantly modulated the size of the Simon effect in the joint Simon task showing an increased Simon effect when the hands of both actors were located near the objects on the monitor, than when they were located away from the monitor. Experiment 2 replicated this finding showing an increased Simon effect when the actor’s hand was located near the objects on the monitor, but only when the co-actor also produced action events in spatial reference. A similar hand position effect was observed in Experiment 3 when a non-human rubber hand replaced the human co-actor. These findings suggest that external action events that are produced in spatial reference bias the distribution of attention to the area near the hand. This strengthens the weight of the spatial response codes (referential coding) and hence increases the joint Simon effect.

Dissociating dynamic probability and predictability in observed actions-an fMRI study

The present fMRI study investigated whether human observers spontaneously exploit the statistical structure underlying continuous action sequences. In particular, we tested whether two different statistical properties can be distinguished with regard to their neural correlates: an action step's predictability and its probability. To assess these properties we used measures from information theory. Predictability of action steps was operationalized by its inverse, conditional entropy, which combines the number of possible action steps with their respective probabilities. Probability of action steps was assessed using conditional surprisal, which increases with decreasing probability. Participants were trained in an action observation paradigm with video clips showing sequences of 9–33 s length with varying numbers of action steps that were statistically structured according to a Markov chain. Behavioral tests revealed that participants implicitly learned this statistical structure, showing that humans are sensitive toward these probabilistic regularities. Surprisal (lower probability) enhanced the BOLD signal in the anterior intraparietal sulcus. In contrast, high conditional entropy, i.e., low predictability, was correlated with higher activity in dorsomedial prefrontal cortex, orbitofrontal gyrus, and posterior intraparietal sulcus. Furthermore, we found a correlation between the anterior hippocampus' response to conditional entropy with the extent of learning, such that the more participants had learnt the structure, the greater the magnitude of hippocampus activation in response to conditional entropy. Findings show that two aspects of predictions can be dissociated: an action's predictability is reflected in a top-down modulation of attentional focus, evident in increased fronto-parietal activation. In contrast, an action's probability depends on the identity of the stimulus itself, resulting in bottom-up driven processing costs in the parietal cortex.

The eye contact effect in request and emblematic hand gestures

Request and emblematic gestures, despite being both communicative gestures, do differ in terms of social valence. Indeed, only the former are used to initiate/maintain/terminate an actual interaction. If such a difference is at stake, a relevant social cue, i.e. eye contact, should have different impacts on the neuronal underpinnings of the two types of gesture. We measured blood oxygen level-dependent signals, using functional magnetic resonance imaging, while participants watched videos of an actor, either blindfolded or not, performing emblems, request gestures, or meaningless control movements. A left-lateralized network was more activated by both types of communicative gestures than by meaningless movements, regardless of the accessibility of the actor's eyes. Strikingly, when eye contact was taken into account as a factor, a right-lateralized network was more strongly activated by emblematic gestures performed by the non-blindfolded actor than by those performed by the blindfolded actor. Such modulation possibly reflects the integration of information conveyed by the eyes with the representation of emblems. Conversely, a wider right-lateralized network was more strongly activated by request gestures performed by the blindfolded than by those performed by the non-blindfolded actor. This probably reflects the effect of the conflict between the observed action and its associated contextual information, in which relevant social cues are missing.

Automatic detection of service initiation signals used in bars

Recognizing the intention of others is important in all social interactions, especially in the service domain. Enabling a bartending robot to serve customers is particularly challenging as the system has to recognize the social signals produced by customers and respond appropriately. Detecting whether a customer would like to order is essential for the service encounter to succeed. This detection is particularly challenging in a noisy environment with multiple customers. Thus, a bartending robot has to be able to distinguish between customers intending to order, chatting with friends or just passing by. In order to study which signals customers use to initiate a service interaction in a bar, we recorded real-life customer-staff interactions in several German bars. These recordings were used to generate initial hypotheses about the signals customers produce when bidding for the attention of bar staff. Two experiments using snapshots and short video sequences then tested the validity of these hypothesized candidate signals. The results revealed that bar staff responded to a set of two non-verbal signals: first, customers position themselves directly at the bar counter and, secondly, they look at a member of staff. Both signals were necessary and, when occurring together, sufficient. The participants also showed a strong agreement about when these cues occurred in the videos. Finally, a signal detection analysis revealed that ignoring a potential order is deemed worse than erroneously inviting customers to order. We conclude that (a) these two easily recognizable actions are sufficient for recognizing the intention of customers to initiate a service interaction, but other actions such as gestures and speech were not necessary, and (b) the use of reaction time experiments using natural materials is feasible and provides ecologically valid results.

Action observers implicitly expect actors to act goal-coherently, even if they do not: an fMRI study

Actions observed in everyday life normally consist of one person performing sequences of goal-directed actions. The present fMRI study tested the hypotheses that observers are influenced by the actor's identity, even when this information is task-irrelevant, and that this information shapes their expectation on subsequent actions of the same actor. Participants watched short video clips of action steps that either pertained to a common action with an overarching goal or not, and were performed by either one or by varying actors (2 × 2 design). Independent of goal coherence, actor coherence elicited activation in dorsolateral and ventromedial frontal cortex, together pointing to a spontaneous attempt to integrate all actions performed by one actor. Interestingly, watching an actor performing unrelated actions elicited additional activation in left inferior frontal gyrus, suggesting a search in semantic memory in an attempt to construct an overarching goal that can reconcile the disparate action steps with a coherent intention. Post-experimental surveys indicate that these processes occur mostly unconsciously. Findings strongly suggest a spontaneous expectation bias toward actor-related episodes in action observers, and hence to the immense impact of actor information on action observation.

N400 ERPs for actions: building meaning in context

Converging neuroscientific evidence suggests the existence of close links between language and sensorimotor cognition. Accordingly, during the comprehension of meaningful actions, our brain would recruit semantic-related operations similar to those associated with the processing of language information. Consistent with this view, electrophysiological findings show that the N400 component, traditionally linked to the semantic processing of linguistic material, can also be elicited by action-related material. This review outlines recent data from N400 studies that examine the understanding of action events. We focus on three specific domains, including everyday action comprehension, co-speech gesture integration, and the semantics involved in motor planning and execution. Based on the reviewed findings, we suggest that both negativities (the N400 and the action-N400) reflect a common neurocognitive mechanism involved in the construction of meaning through the expectancies created by previous experiences and current contextual information. To shed light on how this process is instantiated in the brain, a testable contextual fronto-temporo-parietal model is proposed.

The context-object-manipulation triad: cross talk during action perception revealed by fMRI

To recognize an action, an observer exploits information about the applied manipulation, the involved objects, and the context where the action occurs. Context, object, and manipulation information are hence expected to be tightly coupled in a triadic relationship (the COM triad hereafter). The current fMRI study investigated the hemodynamic signatures of reciprocal modulation in the COM triad. Participants watched short video clips of pantomime actions, that is, actions performed with inappropriate objects, taking place at compatible or incompatible contexts. The usage of pantomime actions enabled the disentanglement of the neural substrates of context-manipulation (CM) and context-object (CO) associations. There were trials in which (1) both manipulation and objects, (2) only manipulation, (3) only objects, or (4) neither manipulation nor objects were compatible with the context. CM compatibility effects were found in an action-related network comprising ventral premotor cortex, SMA, left anterior intraparietal sulcus, and bilateral occipito-temporal cortex. Conversely, CO compatibility effects were found bilaterally in lateral occipital complex. These effects interacted in subregions of the lateral occipital complex. An overlap of CM and CO effects was observed in the occipito-temporal cortex and the dorsal attention network, that is, superior frontal sulcus/dorsal premotor cortex and superior parietal lobe. Results indicate that contextual information is integrated into the analysis of actions. Manipulation and object information is linked by contextual associations as a function of co-occurrence in specific contexts. Activation of either CM or CO associations shifts attention to either action- or object-related relevant information.

Neural changes when actions change: adaptation of strong and weak expectations

Repeated experiences with an event create the expectation that subsequent events will expose an analog structure. These spontaneous expectations rely on an internal model of the event that results from learning. But what happens when events change? Do experience-based internal models get adapted instantaneously, or is model adaptation a function of the solidity of, i.e., familiarity with, the corresponding internal model? The present fMRI study investigated the effects of model solidity on model adaptation in an action observation paradigm. Subjects were made acquainted with a set of action movies that displayed an altered script when encountered again in the scanning session. We found model adaptation to result in an attenuation of the premotor-parietal network for action observation. Model solidity was found to modulate activation in the parahippocampal gyrus and the anterior cerebellar lobules, where increased solidity correlated with activity increase. Finally, the comparison between early and late stages of learning indicated an effect of model solidity on adaptation rate. This contrast revealed the involvement of a fronto-mesial network of Brodmann area 10 and the ACC in those states of learning that were signified by high model solidity, no matter if the memorized original or the altered action model was the more solid component. Findings suggest that the revision of an internal model is dependent on its familiarity. Unwarranted adaptations, but also perseverations may thus be prevented.