Processing of Action and Sound Verbs in Context: An FMRI Study

When abstract becomes concrete, naturalistic encoding of concepts in the brain

Language is acquired and processed in complex and dynamic naturalistic contexts, involving the simultaneous processing of connected speech, faces, bodies, objects, etc. How words and their associated concepts are encoded in the brain during real-world processing is still unknown. Here, the representational structure of concrete and abstract concepts was investigated during movie watching to address the extent to which brain responses dynamically change depending on visual context. First, across contexts, concrete and abstract concepts are shown to encode different experience-based information in separable sets of brain regions. However, these differences are reduced when multimodal context is considered. Specifically, the response profile of abstract words becomes more concrete-like when these are processed in visual scenes highly related to their meaning. Conversely, when the visual context is unrelated to a given concrete word, the activation pattern resembles more that of abstract conceptual processing. These results suggest that while concepts generally encode habitual experiences, the underlying neurobiological organisation is not fixed but depends dynamically on available contextual information.

Conceptual representations in the default, control and attention networks are task-dependent and cross-modal

Conceptual knowledge is central to human cognition. Neuroimaging studies suggest that conceptual processing involves modality-specific and multimodal brain regions in a task-dependent fashion. However, it remains unclear (1) to what extent conceptual feature representations are also modulated by the task, (2) whether conceptual representations in multimodal regions are indeed cross-modal, and (3) how the conceptual system relates to the large-scale functional brain networks. To address these issues, we conducted multivariate pattern analyses on fMRI data. 40 participants performed three tasks-lexical decision, sound judgment, and action judgment-on written words. We found that (1) conceptual feature representations are strongly modulated by the task, (2) conceptual representations in several multimodal regions are cross-modal, and (3) conceptual feature retrieval involves the default, frontoparietal control, and dorsal attention networks. Conceptual representations in these large-scale networks are task-dependent and cross-modal. Our findings support theories that assume conceptual processing to rely on a flexible, multi-level architecture.

The Tensor Brain: A Unified Theory of Perception, Memory, and Semantic Decoding

We present a unified computational theory of an agent's perception and memory. In our model, both perception and memory are realized by different operational modes of the oscillating interactions between a symbolic index layer and a subsymbolic representation layer. The two layers form a bilayer tensor network (BTN). The index layer encodes indices for concepts, predicates, and episodic instances. The representation layer broadcasts information and reflects the cognitive brain state; it is our model of what authors have called the "mental canvas" or the "global workspace." As a bridge between perceptual input and the index layer, the representation layer enables the grounding of indices by their subsymbolic embeddings, which are implemented as connection weights linking both layers. The propagation of activation to earlier perceptual processing layers in the brain can lead to embodiments of indices. Perception and memories first create subsymbolic representations, which are subsequently decoded semantically to produce sequences of activated indices that form symbolic triple statements. The brain is a sampling engine: only activated indices are communicated to the remaining parts of the brain. Triple statements are dynamically embedded in the representation layer and embodied in earlier processing layers: the brain speaks to itself. Although memory appears to be about the past, its main purpose is to support the agent in the present and the future. Recent episodic memory provides the agent with a sense of the here and now. Remote episodic memory retrieves relevant past experiences to provide information about possible future scenarios. This aids the agent in decision making. "Future" episodic memory, based on expected future events, guides planning and action. Semantic memory retrieves specific information, which is not delivered by current perception, and defines priors for future observations. We argue that it is important for the agent to encode individual entities, not just classes and attributes. Perception is learning: episodic memories are constantly being formed, and we demonstrate that a form of self-supervised learning can acquire new concepts and refine existing ones. We test our model on a standard benchmark data set, which we expanded to contain richer representations for attributes, classes, and individuals. Our key hypothesis is that obtaining a better understanding of perception and memory is a crucial prerequisite to comprehending human-level intelligence.

Context matters: How do task demands modulate the recruitment of sensorimotor information during language processing?

Many theories of semantic representation propose that simulations of sensorimotor experience contribute to language processing. This can be seen in the body-object interaction effect (BOI; how easily the human body can interact with a word's referent). Words with high BOI ratings (e.g., ball) are processed more quickly than words with low BOI ratings (e.g., cloud) in various language tasks. This effect can be modulated by task demands. Previous research established that when asked to decide if a word is an object (entity condition), a BOI effect is observed, but when asked to decide if a word is an action (action condition), there is no BOI effect. It is unclear whether the null behavioral effect in the action condition reflects top-down modulation of task-relevant sensorimotor information or the absence of bottom-up activation of sensorimotor simulations. We investigated this question using EEG. In Experiment 1 we replicated the previous behavioral findings. In Experiment 2, 50 participants were assigned to either the entity or action conditions and responded to the same word stimuli. In both conditions we observed differences in ERP components related to the BOI effect. In the entity condition the P2 mean amplitude was significantly more positive for high compared to low BOI words. In the action condition the N400 peak latency was significantly later for high compared to low BOI words. Our findings suggest that BOI information is generated bottom-up regardless of task demands and modulated by top-down processes that recruit sensorimotor information relevant to the task decision.

Effect of the level of task abstraction on the transfer of knowledge from virtual environments in cognitive and motor tasks

Introduction

Virtual environments are increasingly being used for training. It is not fully understood what elements of virtual environments have the most impact and how the virtual training is integrated by the brain on the sought-after skill transference to the real environment. In virtual training, we analyzed how the task level of abstraction modulates the brain activity and the subsequent ability to execute it in the real environment and how this learning generalizes to other tasks. The training of a task under a low level of abstraction should lead to a higher transfer of skills in similar tasks, but the generalization of learning would be compromised, whereas a higher level of abstraction facilitates generalization of learning to different tasks but compromising specific effectiveness.

Methods

A total of 25 participants were trained and subsequently evaluated on a cognitive and a motor task following four training regimes, considering real vs. virtual training and low vs. high task abstraction. Performance scores, cognitive load, and electroencephalography signals were recorded. Transfer of knowledge was assessed by comparing performance scores in the virtual vs. real environment.

Results

The performance to transfer the trained skills showed higher scores in the same task under low abstraction, but the ability to generalize the trained skills was manifested by higher scores under high level of abstraction in agreement with our hypothesis. Spatiotemporal analysis of the electroencephalography revealed higher initial demands of brain resources which decreased as skills were acquired.

Discussion

Our results suggest that task abstraction during virtual training influences how skills are assimilated at the brain level and modulates its manifestation at the behavioral level. We expect this research to provide supporting evidence to improve the design of virtual training tasks.

Meta-analytic evidence for a novel hierarchical model of conceptual processing

Conceptual knowledge plays a pivotal role in human cognition. Grounded cognition theories propose that concepts consist of perceptual-motor features represented in modality-specific perceptual-motor cortices. However, it is unclear whether conceptual processing consistently engages modality-specific areas. Here, we performed an activation likelihood estimation (ALE) meta-analysis across 212 neuroimaging experiments on conceptual processing related to 7 perceptual-motor modalities (action, sound, visual shape, motion, color, olfaction-gustation, and emotion). We found that conceptual processing consistently engages brain regions also activated during real perceptual-motor experience of the same modalities. In addition, we identified multimodal convergence zones that are recruited for multiple modalities. In particular, the left inferior parietal lobe (IPL) and posterior middle temporal gyrus (pMTG) are engaged for three modalities: action, motion, and sound. These "trimodal" regions are surrounded by "bimodal" regions engaged for two modalities. Our findings support a novel model of the conceptual system, according to which conceptual processing relies on a hierarchical neural architecture from modality-specific to multimodal areas up to an amodal hub.

Neural correlates of embodied action language processing: a systematic review and meta-analytic study

The neural correlates of action language processing are still debated within embodied cognition research and little is known about the flexible involvement of modality-specific pre-motor system and multimodal high-level temporo-parietal regions as a function of explicit and implicit tasks. A systematic review and the Activation likelihood estimation (ALE) meta-analyses on functional neuroimaging studies were performed to identify neural correlates of action language processing activated during explicit and implicit tasks. The contrast ALE meta-analysis revealed activation of modality-specific premotor area and inferior frontal areas during explicit action language tasks while a greater activation of posterior temporo-occipital areas emerged for implicit tasks. The conjunction analysis revealed overlap in the temporo-parietal multimodal high-level regions for both types of tasks. Functional specialization of the middle temporal gyrus was found where the more posterior-occipital part resulted activated during implicit action language tasks whereas the antero-lateral part was involved in explicit tasks. Our findings were discussed within a conceptual flexibility perspective about the involvement of both the modality-specific and multimodal brain system during action language processing depending on different types of tasks.

Task-Dependent Functional and Effective Connectivity during Conceptual Processing

Conceptual knowledge is central to cognition. Previous neuroimaging research indicates that conceptual processing involves both modality-specific perceptual-motor areas and multimodal convergence zones. For example, our previous functional magnetic resonance imaging (fMRI) study revealed that both modality-specific and multimodal regions respond to sound and action features of concepts in a task-dependent fashion (Kuhnke P, Kiefer M, Hartwigsen G. 2020b. Task-dependent recruitment of modality-specific and multimodal regions during conceptual processing. Cereb Cortex. 30:3938–3959.). However, it remains unknown whether and how modality-specific and multimodal areas interact during conceptual tasks. Here, we asked 1) whether multimodal and modality-specific areas are functionally coupled during conceptual processing, 2) whether their coupling depends on the task, 3) whether information flows top-down, bottom-up or both, and 4) whether their coupling is behaviorally relevant. We combined psychophysiological interaction analyses with dynamic causal modeling on the fMRI data of our previous study. We found that functional coupling between multimodal and modality-specific areas strongly depended on the task, involved both top-down and bottom-up information flow, and predicted conceptually guided behavior. Notably, we also found coupling between different modality-specific areas and between different multimodal areas. These results suggest that functional coupling in the conceptual system is extensive, reciprocal, task-dependent, and behaviorally relevant. We propose a new model of the conceptual system that incorporates task-dependent functional interactions between modality-specific and multimodal areas.

Images of the unseen: extrapolating visual representations for abstract and concrete words in a data-driven computational model

Theories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants' judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don't have.

Left posterior inferior parietal cortex causally supports the retrieval of action knowledge

Conceptual knowledge is central to human cognition. The left posterior inferior parietal lobe (pIPL) is implicated by neuroimaging studies as a multimodal hub representing conceptual knowledge related to various perceptual-motor modalities. However, the causal role of left pIPL in conceptual processing remains unclear. Here, we transiently disrupted left pIPL function with transcranial magnetic stimulation (TMS) to probe its causal relevance for the retrieval of action and sound knowledge. We compared effective TMS over left pIPL with sham TMS, while healthy participants performed three different tasks-lexical decision, action judgment, and sound judgment-on words with a high or low association to actions and sounds. We found that pIPL-TMS selectively impaired action judgments on low sound-low action words. For the first time, we directly related computational simulations of the TMS-induced electrical field to behavioral performance, which revealed that stronger stimulation of left pIPL is associated with worse performance for action but not sound judgments. These results indicate that left pIPL causally supports conceptual processing when action knowledge is task-relevant and cannot be compensated by sound knowledge. Our findings suggest that left pIPL is specialized for the retrieval of action knowledge, challenging the view of left pIPL as a multimodal conceptual hub.

Time course of brain activity during the processing of motor- and vision-related abstract concepts: flexibility and task dependency

Grounded cognition theories assume that conceptual processing depends on modality-specific brain systems in a context-dependent fashion. Although the relation of abstract concepts to modality-specific systems is less obvious than for concrete concepts, recent behavioral and neuroimaging studies indicated a foundation of abstract concepts in vision and action. However, due to their poor temporal resolution, neuroimaging studies cannot determine whether sensorimotor activity reflects rapid access to conceptual information or later conceptual processes. The present study therefore assessed the time course of abstract concept processing using event-related potentials (ERPs) and compared ERP responses to abstract concepts with a strong relation to vision or action. We tested whether possible ERP effects to abstract word categories would emerge in early or in later time windows and whether these effects would depend on the depth of the conceptual task. In Experiment 1, a shallow lexical decision task, early feature-specific effects starting at 178 ms were revealed, but later effects beyond 300 ms were also observed. In Experiment 2, a deep conceptual decision task, feature-specific effects with an onset of 22 ms were obtained, but effects again extended beyond 300 ms. In congruency with earlier neuroimaging work, the present feature-specific ERP effects suggest a grounding of abstract concepts in modal brain systems. The presence of early and late feature-specific effects indicates that sensorimotor activity observed in neuroimaging experiments may reflect both rapid conceptual and later post-conceptual processing. Results furthermore suggest that a deep conceptual task accelerates access to conceptual sensorimotor features, thereby demonstrating conceptual flexibility.

Brain Activation During Conceptual Processing of Action and Sound Verbs

Grounded cognition approaches to conceptual representations postulate a close link between conceptual knowledge and the sensorimotor brain systems. The present fMRI study tested, whether a feature-specific representation of concepts, as previously demonstrated for nouns, can also be found for action- and sound-related verbs. Participants were presented with action- and soundrelated verbs along with pseudoverbs while performing a lexical decision task. Sound-related verbs activated auditory areas in the temporal cortex, whereas action-related verbs activated brain regions in the superior frontal gyrus and the cerebellum, albeit only at a more liberal threshold. This differential brain activation during conceptual verb processing partially overlapped with or was adjacent to brain regions activated during the functional localizers probing sound perception or action execution. Activity in brain areas involved in the processing of action information was parametrically modulated by ratings of action relevance. Comparisons of action- and sound-related verbs with pseudoverbs revealed activation for both verb categories in auditory and motor areas. In contrast to proposals of strong grounded cognition approaches, our study did not demonstrate a considerable overlap of activations for action- and sound-related verbs and for the corresponding functional localizer tasks. However, in line with weaker variants of grounded cognition theories, the differential activation pattern for action- and sound-related verbs was near corresponding sensorimotor brain regions depending on conceptual feature relevance. Possibly, action-sound coupling resulted in a mutual activation of the motor and the auditory system for both action- and sound-related verbs, thereby reducing the effect sizes for the differential contrasts.

Task-Dependent Recruitment of Modality-Specific and Multimodal Regions during Conceptual Processing

Conceptual knowledge is central to cognitive abilities such as word comprehension. Previous neuroimaging evidence indicates that concepts are at least partly composed of perceptual and motor features that are represented in the same modality-specific brain regions involved in actual perception and action. However, it is unclear to what extent the retrieval of perceptual-motor features and the resulting engagement of modality-specific regions depend on the concurrent task. To address this issue, we measured brain activity in 40 young and healthy participants using functional magnetic resonance imaging, while they performed three different tasks-lexical decision, sound judgment, and action judgment-on words that independently varied in their association with sounds and actions. We found neural activation for sound and action features of concepts selectively when they were task-relevant in brain regions also activated during auditory and motor tasks, respectively, as well as in higher-level, multimodal regions which were recruited during both sound and action feature retrieval. For the first time, we show that not only modality-specific perceptual-motor areas but also multimodal regions are engaged in conceptual processing in a flexible, task-dependent fashion, responding selectively to task-relevant conceptual features.