The Representation of Observed Actions at the Subordinate, Basic, and Superordinate Level

Low-frequency neural activity tracks syntactic information through semantic mediation

How our brain integrates single words into larger linguistic units is a central focus in neurolinguistic studies. Previous studies mainly explored this topic at the semantic or syntactic level, with few looking at how cortical activities track word sequences with different levels of semantic correlations. In addition, prior research did not tease apart the semantic factors from the syntactic ones in the word sequences. The current study addressed these issues by conducting a speech perception EEG experiment using the frequency-tagging paradigm. Participants (N = 25, Meanage = 23;4, 16 girls) were asked to listen to different types of sequences and their neural activity was recorded by EEG. We also constructed a model simulation based on surprisal values of GPT-2. Both the EEG results and the model prediction show that low-frequency neural activity tracks syntactic information through semantic mediation. Implications of the findings were discussed in relation to the language processing mechanism.

Shared representations of human actions across vision and language

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Overlapping representations of observed actions and action-related features

The lateral occipitotemporal cortex (LOTC) has been shown to capture the representational structure of a smaller range of actions. In the current study, we carried out an fMRI experiment in which we presented human participants with images depicting 100 different actions and used representational similarity analysis (RSA) to determine which brain regions capture the semantic action space established using judgments of action similarity. Moreover, to determine the contribution of a wide range of action-related features to the neural representation of the semantic action space we constructed an action feature model on the basis of ratings of 44 different features. We found that the semantic action space model and the action feature model are best captured by overlapping activation patterns in bilateral LOTC and ventral occipitotemporal cortex (VOTC). An RSA on eight dimensions resulting from principal component analysis carried out on the action feature model revealed partly overlapping representations within bilateral LOTC, VOTC, and the parietal lobe. Our results suggest spatially overlapping representations of the semantic action space of a wide range of actions and the corresponding action-related features. Together, our results add to our understanding of the kind of representations along the LOTC that support action understanding.