The context-object-manipulation triad: cross talk during action perception revealed by fMRI

Shared representations of human actions across vision and language

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Differential contributions of body form, motion, and temporal information to subjective action understanding in naturalistic stimuli

Introduction

We investigated the factors underlying naturalistic action recognition and understanding, as well as the errors occurring during recognition failures.

Methods

Participants saw full-light stimuli of ten different whole-body actions presented in three different conditions: as normal videos, as videos with the temporal order of the frames scrambled, and as single static representative frames. After each stimulus presentation participants completed one of two tasks-a forced choice task where they were given the ten potential action labels as options, or a free description task, where they could describe the action performed in each stimulus in their own words.

Results

While generally, a combination of form, motion, and temporal information led to the highest action understanding, for some actions form information was sufficient and adding motion and temporal information did not increase recognition accuracy. We also analyzed errors in action recognition and found primarily two different types.

Discussion

One type of error was on the semantic level, while the other consisted of reverting to the kinematic level of body part processing without any attribution of semantics. We elaborate on these results in the context of naturalistic action perception.

The pain hidden in your hands: Facial expression of pain reduces the influence of goal-related information in action recognition

The involvement of the sensorimotor system in the perception of painful actions has been repeatedly demonstrated. Yet the cognitive processes corresponding to sensorimotor activations have not been identified. In particular, the respective role of higher-level and lower-level action representations such as goals and grips in the recognition of painful actions is not clear. Previous research has shown that in a neutral context, higher-level action representations (goals) are prioritized over lower-level action representations (grips) and guide action recognition. The present study evaluates to what extent the general priority given to goal-related information in the processing of visual actions can be modulated by a context of pain. We used the action violation paradigm developed by van Elk et al. (2008). In the present action tasks, participants had to judge whether the grip or the goal of object-directed actions displayed in photographs was correct or not. The actress in the photograph could show either a neutral facial expression or a facial expression of pain. In the control task, they had to judge whether the actress expressed pain. In the action tasks, goals influenced grip judgements more than grips influenced goal judgements overall, corroborating the priority given to goal-related information previously reported. Critically, the impact of irrelevant goal-related information on the identification of incorrect grips disappeared in the pain context. Moreover, judgements in the control task were similarly influenced by grip and goal-related information. Results suggest that a context of pain reduces the reliance on higher-level action for action judgments. Findings provide novel directions regarding the cognitive and brain mechanisms involved in action processing in painful situations and support pluralist views of action understanding.

The cognitive structure underlying the organization of observed actions

In daily life, we frequently encounter actions performed by other people. Here we aimed to examine the key categories and features underlying the organization of a wide range of actions in three behavioral experiments (N = 378 participants). In Experiment 1, we used a multi-arrangement task of 100 different actions. Inverse multidimensional scaling and hierarchical clustering revealed 11 action categories, including Locomotion, Communication, and Aggressive actions. In Experiment 2, we used a feature-listing paradigm to obtain a wide range of action features that were subsequently reduced to 59 key features and used in a rating study (Experiment 3). A direct comparison of the feature ratings obtained in Experiment 3 between actions belonging to the categories identified in Experiment 1 revealed a number of features that appear to be critical for the distinction between these categories, e.g., the features Harm and Noise for the category Aggressive actions, and the features Targeting a person and Contact with others for the category Interaction. Finally, we found that a part of the category-based organization is explained by a combination of weighted features, whereas a significant proportion of variability remained unexplained, suggesting that there are additional sources of information that contribute to the categorization of observed actions. The characterization of action categories and their associated features serves as an important extension of previous studies examining the cognitive structure of actions. Moreover, our results may serve as the basis for future behavioral, neuroimaging and computational modeling studies.

Two distinct networks containing position-tolerant representations of actions in the human brain

Humans can recognize others' actions in the social environment. This action recognition ability is rarely hindered by the movement of people in the environment. The neural basis of this position tolerance for observed actions is not fully understood. Here, we aimed to identify brain regions capable of generalizing representations of actions across different positions and investigate the representational content of these regions. In a functional magnetic resonance imaging experiment, participants viewed point-light displays of different human actions. Stimuli were presented in either the upper or the lower visual field. Multivariate pattern analysis and a surface-based searchlight approach were employed to identify brain regions that contain position-tolerant action representation: Classifiers were trained with patterns in response to stimuli presented in one position and were tested with stimuli presented in another position. Results showed above-chance classification in the left and right lateral occipitotemporal cortices, right intraparietal sulcus, and right postcentral gyrus. Further analyses exploring the representational content of these regions showed that responses in the lateral occipitotemporal regions were more related to subjective judgments, while those in the parietal regions were more related to objective measures. These results provide evidence for two networks that contain abstract representations of human actions with distinct representational content.

Social-affective features drive human representations of observed actions

Humans observe actions performed by others in many different visual and social settings. What features do we extract and attend when we view such complex scenes, and how are they processed in the brain? To answer these questions, we curated two large-scale sets of naturalistic videos of everyday actions and estimated their perceived similarity in two behavioral experiments. We normed and quantified a large range of visual, action-related, and social-affective features across the stimulus sets. Using a cross-validated variance partitioning analysis, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action features in both behavioral experiments. Next, we conducted an electroencephalography experiment, which revealed a sustained correlation between neural responses to videos and their behavioral similarity. Visual, action, and social-affective features predicted neural patterns at early, intermediate, and late stages, respectively, during this behaviorally relevant time window. Together, these findings show that social-affective features are important for perceiving naturalistic actions and are extracted at the final stage of a temporal gradient in the brain.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

People accurately predict the transition probabilities between actions

Social life is a complex dance. To coordinate gracefully with one's partners, one must predict their actions. Here, we investigated how people predict others' actions. We hypothesized that people can accurately predict others' future actions based on knowledge of their current actions, coupled with knowledge of action transitions. To test whether people have accurate knowledge of the transition probabilities between actions, we compared actual rates of action transitions-calculated from four large naturalistic datasets-to participants' ratings of the transition probabilities between corresponding sets of actions. In five preregistered studies, participants demonstrated accurate mental models of action transitions. Furthermore, we found that people drew upon conceptual knowledge of actions-described by the six-dimensional ACT-FASTaxonomy-to guide their accurate predictions. Together, these results indicate that people can accurately anticipate other people's moves in the dance of social life and that the structure of action knowledge may be tailored to making these predictions.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Predicting other people’s upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person’s identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action’s duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects’ touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions’ goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

Possible association between the integrity of cerebellar pathways and neurocognitive performance in children with posterior fossa tumors

BACKGROUND

Cerebellar tumor survivors often exhibit neuropsychological deficits that could be related to alterations in cerebro-cerebellar networks. This is a pilot study designed to understand if diffusion tensor imaging (DTI)-based tractography is able to identify possible correlations between cerebellar white matter structure and cognitive outcome in children on long-term follow-up for posterior fossa (PF) tumors who were thoroughly assessed for neuropsychological functioning.

METHODS

DTI-based tractography was performed in pediatric patients with PF tumors. Fractional anisotropy (FA) and volumetric measurements of spinocerebellar, dentorubrothalamocortical and corticopontocerebellar tracts were analyzed. Cognitive and neuropsychological functioning was assessed by the Wechsler Intelligence Scale for Children-IV Edition (WISC-IV) and the Developmental Neuropsychological Assessment (NEPSY II). The associations between Full-Scale Intelligence Quotient (FSIQ), NEPSY-II scores, and fiber tracts were tested by the Spearman rank correlation coefficient.

RESULTS

Seven patients (median age at diagnosis five years, range, 3-13) treated for medulloblastoma (2/7; 29%) and pilocytic astrocytoma (5/7; 71%) were retrospectively evaluated. All children had complete surgery. The median FSIQ was 84 (range, 67-93). Patients presented with several deficits on many NEPSY-II tasks; in particular, memory was impaired in nearly half of them. FSIQ and neurocognitive tasks significantly correlated with specific corticopontocerebellar tracts.

CONCLUSION

Children on follow-up for PF tumor showed scattered cognitive impairments, including deficits in long-term and immediate memory. Tractography allowed us to describe a possible association between the integrity of cerebellar pathways and neurocognitive performance, suggesting that the myelinization of these fibers may represent an indicator for the development of long-term cognitive sequelae.

Predictive Impact of Contextual Objects during Action Observation: Evidence from Functional Magnetic Resonance Imaging

The processing of congruent stimuli, such as an object or action in its typical location, is usually associated with reduced neural activity, probably due to facilitated recognition. However, in some situations, congruency increases neural activity—for example, when objects next to observed actions are likely versus unlikely to be involved in forthcoming action steps. Here, we investigated using fMRI whether the processing of contextual cues during action perception is driven by their (in)congruency and, thus, informative value to make sense of an observed scene. Specifically, we tested whether both highly congruent contextual objects (COs), which strongly indicate a future action step, and highly incongruent COs, which require updating predictions about possible forthcoming action steps, provide more anticipatory information about the action course than moderately congruent COs. In line with our hypothesis that especially the inferior frontal gyrus (IFG) subserves the integration of the additional information into the predictive model of the action, we found highly congruent and incongruent COs to increase bilateral activity in action observation nodes, that is, the IFG, the occipitotemporal cortex, and the intraparietal sulcus. Intriguingly, BA 47 was significantly stronger engaged for incongruent COs reflecting the updating of prediction in response to conflicting information. Our findings imply that the IFG reflects the informative impact of COs on observed actions by using contextual information to supply and update the currently operating predictive model. In the case of an incongruent CO, this model has to be reconsidered and extended toward a new overarching action goal.

Observation of an action with a congruent contextual background facilitates corticospinal excitability: A combined TMS and eye-tracking experiment

Action observation produces activity in similar regions of the brain to those involved in action execution and can offer an effective intervention for motor (re)learning, although optimal viewing conditions for such interventions remain to be established. In this experiment, single-pulse transcranial magnetic stimulation (TMS) and eye-tracking were used simultaneously to investigate the effect of manipulating background context on both corticospinal excitability and visual attention during action observation. Twenty-four participants observed four different videos: (i) a static hand holding a sponge (control condition); and an index finger-thumb pinch of a sponge against (ii) a plain black background; (iii) a background containing objects that were incongruent with the observed action; and (iv) a background containing objects that were congruent with the observed action. TMS was delivered to the hand representation of the left primary motor cortex, and motor evoked potentials were recorded from the first dorsal interosseous and abductor digiti minimi muscles of the right hand. Eye movements were recorded throughout the experiment. Results indicated that corticospinal excitability was facilitated during the congruent context condition compared to both the static hand and plain black background conditions. In addition, the number of fixations and percentage of time participants spent fixating on the background scene were significantly greater during the incongruent and congruent conditions compared to the static hand and plain black background conditions. These results indicate that the provision of additional visual information that is congruent with the observed movement contributes to a facilitation of corticospinal excitability by providing the observer with information regarding the goal and intention of the observed action. Providing congruent contextual information may enhance the efficacy of action observation interventions for motor (re)learning.

Making sense of objects lying around: How contextual objects shape brain activity during action observation

Action recognition involves not only the readout of body movements and involved objects but also the integration of contextual information, e.g. the environment in which an action takes place. Notably, inferring superordinate goals and generating predictions about forthcoming action steps should benefit from screening the actor's immediate environment, in particular objects located in the actor's peripersonal space and thus potentially used in following action steps. Critically, if such contextual objects (COs) afford actions that are semantically related to the observed action, they may trigger or facilitate the inference of goals and the prediction of following actions. This fMRI study investigated the neural mechanisms underlying the integration of COs in semantic and spatial relation to observed actions. Specifically, we tested the hypothesis that the inferior frontal gyrus (IFG) subserves this integration. Participants observed action videos in which COs and observed actions had common overarching goals or not (goal affinity) and varied in their location relative to the actor. High goal affinity increased bilateral activity in action observation network nodes, i.e. the occipitotemporal cortex and the intraparietal sulcus, but also in the precuneus and middle frontal gyri. This finding suggests that the semantic relation between COs and actions is considered during action observation and triggers (rather than facilitates) processes beyond those usually involved in action observation. Moreover, COs with high goal affinity located close to the actor's dominant hand additionally engaged bilateral IFG, corroborating the view that IFG is critically involved in the integration of action steps under a common overarching goal.

Visual context modulates potentiation of grasp types during semantic object categorization

Substantial evidence suggests that conceptual processing of manipulable objects is associated with potentiation of action. Such data have been viewed as evidence that objects are recognized via access to action features. Many objects, however, are associated with multiple actions. For example, a kitchen timer may be clenched with a power grip to move it but pinched with a precision grip to use it. The present study tested the hypothesis that action evocation during conceptual object processing is responsive to the visual scene in which objects are presented. Twenty-five healthy adults were asked to categorize object pictures presented in different naturalistic visual contexts that evoke either move- or use-related actions. Categorization judgments (natural vs. artifact) were performed by executing a move- or use-related action (clench vs. pinch) on a response device, and response times were assessed as a function of contextual congruence. Although the actions performed were irrelevant to the categorization judgment, responses were significantly faster when actions were compatible with the visual context. This compatibility effect was largely driven by faster pinch responses when objects were presented in use-compatible, as compared with move-compatible, contexts. The present study is the first to highlight the influence of visual scene on stimulus-response compatibility effects during semantic object processing. These data support the hypothesis that action evocation during conceptual object processing is biased toward context-relevant actions.

How the context matters. Literal and figurative meaning in the embodied language paradigm

The involvement of the sensorimotor system in language understanding has been widely demonstrated. However, the role of context in these studies has only recently started to be addressed. Though words are bearers of a semantic potential, meaning is the product of a pragmatic process. It needs to be situated in a context to be disambiguated. The aim of this study was to test the hypothesis that embodied simulation occurring during linguistic processing is contextually modulated to the extent that the same sentence, depending on the context of utterance, leads to the activation of different effector-specific brain motor areas. In order to test this hypothesis, we asked subjects to give a motor response with the hand or the foot to the presentation of ambiguous idioms containing action-related words when these are preceded by context sentences. The results directly support our hypothesis only in relation to the comprehension of hand-related action sentences.

The neural correlates of planning and executing actual tool use

Human tool use is complex, and underlying neural mechanisms seem to be widely distributed across several brain systems; however, neuroimaging studies of actual tool use are rare because of experimental challenges hindering detailed analysis within one acting subject. We developed a "Tool-Carousel" that enabled us to test actual manipulation of different objects during fMRI and investigate the planning and execution of goal-directed actions. Particularly, we focused on the effects of three factors on object manipulations: the type of object manipulated, the type of manipulation, and the hand to be used. The main focus lay on the question of how complex object use compared with unspecific actions are processed and especially how such representations interact with the knowledge about the object in the action-related dorsal stream. We found that object manipulations with both right and left hand recruit a common network strongly lateralized to the left hemisphere especially during planning but also action execution. Specifically, while activity in the ventral stream was involved in processing semantic information and object properties, a dorso-dorsal pathway (i.e., superior occipital gyrus, superior parietal lobule, and dorsal premotor area) was relevant for monitoring the online control of objects and also a ventro-dorsal pathway (i.e., middle occipital gyrus, inferior parietal lobule, and ventral premotor area) was specifically involved in processing known object manipulations, such as tool use. Data further indicate an interaction of ventral stream areas, such as middle temporal gyrus and lateral occipital complex, with both dorsal pathways. These results provide evidence for left-lateralized occipito-temporo-parieto-frontal network of everyday tool use, which may help to characterize specific deficits in patients suffering from apraxia.

The tool in the brain: apraxia in ADL. Behavioral and neurological correlates of apraxia in daily living

Humans differ from other animals in the way they can skilfully and precisely operate or invent tools to facilitate their everyday life. Tools have dominated our home, travel and work environment, becoming an integral step for our motor skills development. What happens when the part of the brain responsible for tool use is damaged in our adult life due to a cerebrovascular accident? How does daily life change when we lose the previously mastered ability to make use of the objects around us? How do patients suffering from compromised tool use cope with food preparation, personal hygiene, grooming, housework, or use of home appliances? In this literature review we present a state of the art for single and multiple tool use research, with a focus on the impact that apraxia (impaired ability to perform tool-based actions) and action disorganization syndrome (ADS; impaired ability to carry out multi-step actions) have on activities of daily living (ADL). Firstly, we summarize the behavioral studies investigating the impact of apraxia and other comorbidity syndromes, such as neglect or visual extinction, on ADL. We discuss the hallmarks of the compromised tool use in terms of the sequencing of action steps, conceptual errors committed, spatial motor control, and temporal organization of the movement. In addition, we present an up-to-date overview of the neuroimaging and lesion analyses studies that provide an insight into neural correlates of tool use in the human brain and functional changes in the neural organization following a stroke, in the context of ADL. Finally we discuss the current practice in neurorehabilitation of ADL in apraxia and ADS aiming at increasing patients' independence.

N400 ERPs for actions: building meaning in context

Converging neuroscientific evidence suggests the existence of close links between language and sensorimotor cognition. Accordingly, during the comprehension of meaningful actions, our brain would recruit semantic-related operations similar to those associated with the processing of language information. Consistent with this view, electrophysiological findings show that the N400 component, traditionally linked to the semantic processing of linguistic material, can also be elicited by action-related material. This review outlines recent data from N400 studies that examine the understanding of action events. We focus on three specific domains, including everyday action comprehension, co-speech gesture integration, and the semantics involved in motor planning and execution. Based on the reviewed findings, we suggest that both negativities (the N400 and the action-N400) reflect a common neurocognitive mechanism involved in the construction of meaning through the expectancies created by previous experiences and current contextual information. To shed light on how this process is instantiated in the brain, a testable contextual fronto-temporo-parietal model is proposed.