Empirically grounding grounded cognition: the case of color

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Context effects in language comprehension: The role of emotional state and attention on semantic and syntactic processing

Semantics and syntax are core components of language. The prevailing view was that processing of word meaning and syntactic processing happens in isolation from other systems. In light of proofed interactions between language and other systems, especially with perception, action and emotion, this view became untenable. This article reviews Event-related potential studies conducted at the Donders Centre for Cognition exploring the interplay between language comprehension and a person's emotional state. The research program was aimed at an investigation of the online effects of emotional state on semantic processing and syntactic processing. To this aim we manipulated mood via film fragments (happy vs. sad) before participants read neutral sentences while their EEG was recorded. In Part 1, it is shown that mood impacts online semantic processing (as indicated by N400) and the processing of syntactic violations (as indicated by P600). Part 2 was directed at a further determination of the mechanisms underlying these interactions. The role of heuristics was examined by investigating the effects of mood on the P600 to semantic reversals. The results revealed that mood affects heuristic processing. The next step consisted of an assessment of the role of attention, in the mood-by-semantics and mood-by-syntax interaction. This was accomplished by recording EEG while manipulating attention via task next to emotional state. Participants performed a semantic or syntactic judgment task vs. a letter-size judgment task. The main ERP results were as follows: (i) attention interacts with the mood effect on semantic processing and syntactic processing, respectively, (ii) the effects of mood on semantic processing and syntactic processing are reliable, and (iii) the mood effects on semantic processing are not fixed but context-dependent. In Part 3 the effects of mood on the processing of script knowledge and general world knowledge are presented. Part 4 closes with a discussion of the mechanisms involved in the mood-by-language interactions and recommendations for future research. Regarding the underlying mechanism we propose that heuristics based on semantic expectancies or syntactic expectancies play a key role in the mood-by-language interactions. The results support the view that language takes place in continuous interaction with other (non-language) systems.

No fruits without color: Cross-modal priming and EEG reveal different roles for different features across semantic categories

Category-specific impairments witnessed in patients with semantic deficits have broadly dissociated into natural and artificial kinds. However, how the category of food (more specifically, fruits and vegetables) fits into this distinction has been difficult to interpret, given a pattern of deficit that has inconsistently mapped onto either kind, despite its intuitive membership to the natural domain. The present study explores the effects of a manipulation of a visual sensory (i.e., color) or functional (i.e., orientation) feature on the consequential semantic processing of fruits and vegetables (and tools, by comparison), first at the behavioral and then at the neural level. The categorization of natural (i.e., fruits/vegetables) and artificial (i.e., utensils) entities was investigated via cross-modal priming. Reaction time analysis indicated a reduction in priming for color-modified natural entities and orientation-modified artificial entities. Standard event-related potentials (ERP) analysis was performed, in addition to linear classification. For natural entities, a N400 effect at central channel sites was observed for the color-modified condition compared relative to normal and orientation conditions, with this difference confirmed by classification analysis. Conversely, there was no significant difference between conditions for the artificial category in either analysis. These findings provide strong evidence that color is an integral property to the categorization of fruits/vegetables, thus substantiating the claim that feature-based processing guides as a function of semantic category.

Making It Harder to "See" Meaning: The More You See Something, the More Its Conceptual Representation Is Susceptible to Visual Interference

Does the perceptual system for looking at the world overlap with the conceptual system for thinking about it? We conducted two experiments (N = 403) to investigate this question. Experiment 1 showed that when people make simple semantic judgments on words, interference from a concurrent visual task scales in proportion to how much visual experience they have with the things the words refer to. Experiment 2 showed that when people make the same judgments on the very same words, interference from a concurrent manual task scales in proportion to how much manual (but critically, not visual) experience people have with those same things. These results suggest that the meanings of frequently visually experienced things are represented (in part) in the visual system used for actually seeing them, that this visually represented information is a functional part of conceptual knowledge, and that the extent of these visual representations is influenced by visual experience.

Imagined event files: An interplay between imagined and perceived objects

An important function of attention is to integrate features processed in distinct brain areas into a single coherent object representation. The immediate outcome of this binding process has been termed an event file, a transient memory structure that links features, context, and associated actions. A key result that supports the existence of event files is the partial repetition cost - slowed responses to a current event thought to reflect the updating of event file bindings in simple trial-to-trial repetition methods. In four experiments, using a procedure similar to Hommel (Visual Cognition, 5 (1/2), 183-216, 1998), we explored whether similar event file binding effects occurred when participants imagine rather than perceive a first event prior to responding to a following visual event. The results indicate that this effect does occur, implying that feature binding in imagery and perception may follow similar principles.

Semantic Interference and Facilitation: Understanding the Integration of Spatial Distance and Conceptual Similarity During Sentence Reading

Existing evidence has shown a processing advantage (or facilitation) when representations derived from a non-linguistic context (spatial proximity depicted by gambling cards moving together) match the semantic content of an ensuing sentence. A match, inspired by conceptual metaphors such as 'similarity is closeness' would, for instance, involve cards moving closer together and the sentence relates similarity between abstract concepts such as war and battle. However, other studies have reported a disadvantage (or interference) for congruence between the semantic content of a sentence and representations of spatial distance derived from this sort of non-linguistic context. In the present article, we investigate the cognitive mechanisms underlying the interaction between the representations of spatial distance and sentence processing. In two eye-tracking experiments, we tested the predictions of a mechanism that considers the competition, activation, and decay of visually and linguistically derived representations as key aspects in determining the qualitative pattern and time course of that interaction. Critical trials presented two playing cards, each showing a written abstract noun; the cards turned around, obscuring the nouns, and moved either farther apart or closer together. Participants then read a sentence expressing either semantic similarity or difference between these two nouns. When instructed to attend to the nouns on the cards (Experiment 1), participants' total reading times revealed interference between spatial distance (e.g., closeness) and semantic relations (similarity) as soon as the sentence explicitly conveyed similarity. But when instructed to attend to the cards (Experiment 2), cards approaching (vs. moving apart) elicited first interference (when similarity was implicit) and then facilitation (when similarity was made explicit) during sentence reading. We discuss these findings in the context of a competition mechanism of interference and facilitation effects.

Projectors, associators, visual imagery, and the time course of visual processing in grapheme-color synesthesia

In grapheme-color synesthesia, seeing particular letters or numbers evokes the experience of specific colors. We investigate the brain's real-time processing of words in this population by recording event-related brain potentials (ERPs) from 15 grapheme-color synesthetes and 15 controls as they judged the validity of word pairs ('yellow banana' vs. 'blue banana') presented under high and low visual contrast. Low contrast words elicited delayed P1/N170 visual ERP components in both groups, relative to high contrast. When color concepts were conveyed to synesthetes by individually tailored achromatic grapheme strings ('55555 banana'), visual contrast effects were like those in color words: P1/N170 components were delayed but unchanged in amplitude. When controls saw equivalent colored grapheme strings, visual contrast modulated P1/N170 amplitude but not latency. Color induction in synesthetes thus differs from color perception in controls. Independent from experimental effects, all orthographic stimuli elicited larger N170 and P2 in synesthetes than controls. While P2 (150-250ms) enhancement was similar in all synesthetes, N170 (130-210ms) amplitude varied with individual differences in synesthesia and visual imagery. Results suggest immediate cross-activation in visual areas processing color and shape is most pronounced in so-called projector synesthetes whose concurrent colors are experienced as originating in external space.

GRAPES-Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain

In this article, I discuss some of the latest functional neuroimaging findings on the organization of object concepts in the human brain. I argue that these data provide strong support for viewing concepts as the products of highly interactive neural circuits grounded in the action, perception, and emotion systems. The nodes of these circuits are defined by regions representing specific object properties (e.g., form, color, and motion) and thus are property-specific, rather than strictly modality-specific. How these circuits are modified by external and internal environmental demands, the distinction between representational content and format, and the grounding of abstract social concepts are also discussed.

Grounding grammatical categories: attention bias in hand space influences grammatical congruency judgment of Chinese nominal classifiers

Embodied cognitive theories predict that linguistic conceptual representations are grounded and continually represented in real world, sensorimotor experiences. However, there is an on-going debate on whether this also holds for abstract concepts. Grammar is the archetype of abstract knowledge, and therefore constitutes a test case against embodied theories of language representation. Former studies have largely focussed on lexical-level embodied representations. In the present study we take the grounding-by-modality idea a step further by using reaction time (RT) data from the linguistic processing of nominal classifiers in Chinese. We take advantage of an independent body of research, which shows that attention in hand space is biased. Specifically, objects near the hand consistently yield shorter RTs as a function of readiness for action on graspable objects within reaching space, and the same biased attention inhibits attentional disengagement. We predicted that this attention bias would equally apply to the graspable object classifier but not to the big object classifier. Chinese speakers (N = 22) judged grammatical congruency of classifier-noun combinations in two conditions: graspable object classifier and big object classifier. We found that RTs for the graspable object classifier were significantly faster in congruent combinations, and significantly slower in incongruent combinations, than the big object classifier. There was no main effect on grammatical violations, but rather an interaction effect of classifier type. Thus, we demonstrate here grammatical category-specific effects pertaining to the semantic content and by extension the visual and tactile modality of acquisition underlying the acquisition of these categories. We conclude that abstract grammatical categories are subjected to the same mechanisms as general cognitive and neurophysiological processes and may therefore be grounded.

The combined effects of neurostimulation and priming on creative thinking. A preliminary tDCS study on dorsolateral prefrontal cortex

The role of prefrontal cortex (PFC) in influencing creative thinking has been investigated by many researchers who, while succeeding in proving an effective involvement of PFC, reported suggestive but sometimes conflicting results. In order to better understand the relationships between creative thinking and brain activation in a more specific area of the PFC, we explored the role of dorsolateral PFC (DLPFC). We devised an experimental protocol using transcranial direct-current stimulation (tDCS). The study was based on a 3 (kind of stimulation: anodal vs. cathodal vs. sham) × 2 (priming: divergent vs. convergent) design. Forty-five healthy adults were randomly assigned to one stimulation condition. Participants' creativity skills were assessed using the Product Improvement subtest from the Torrance Tests of Creative Thinking (TTCT). After 20 min of tDCS stimulation, participants were presented with visual images of common objects. Half of the participants were instructed to visualize themselves using the object in an unusual way (divergent priming), whereas the other half were asked to visualize themselves while using the object in a common way (convergent priming). Priming was aimed at inducing participants to adopt different attitudes toward the creative task. Afterwards, participants were asked to describe all of the possible uses of the objects that were presented. Participants' physiological activation was recorded using a biofeedback equipment. Results showed a significant effect of anodal stimulation that enhanced creative performance, but only after divergent priming. Participants showed lower skin temperature values after cathodal stimulation, a finding which is coherent with studies reporting that, when a task is not creative or creative thinking is not prompted, people show lower levels of arousal. Differences in individual levels of creativity as assessed by the Product Improvement test were not influential. The involvement of DLPFC in creativity has been supported, presumably in association to shift of attention modulated by priming.