Concrete vs. Abstract Semantics: From Mental Representations to Functional Brain Mapping

Language with vision: A study on grounded word and sentence embeddings

Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations by incorporating perceptual knowledge from vision into text-based representations. Despite many attempts at language grounding, achieving an optimal equilibrium between textual representations of the language and our embodied experiences remains an open field. Some common concerns are the following. Is visual grounding advantageous for abstract words, or is its effectiveness restricted to concrete words? What is the optimal way of bridging the gap between text and vision? To what extent is perceptual knowledge from images advantageous for acquiring high-quality embeddings? Leveraging the current advances in machine learning and natural language processing, the present study addresses these questions by proposing a simple yet very effective computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information while simultaneously preserving the distributional statistics that characterize word usage in text corpora. By applying a learned alignment, we are able to indirectly ground unseen words including abstract words. A series of evaluations on a range of behavioral datasets shows that visual grounding is beneficial not only for concrete words but also for abstract words, lending support to the indirect theory of abstract concepts. Moreover, our approach offers advantages for contextualized embeddings, such as those generated by BERT (Devlin et al, 2018), but only when trained on corpora of modest, cognitively plausible sizes. Code and grounded embeddings for English are available at ( https://github.com/Hazel1994/Visually_Grounded_Word_Embeddings_2 ).

Aberrant brain language network in schizophrenia spectrum disorder: a systematic review of its relation to language signs beyond symptoms

Background

Language disturbances are a core feature of schizophrenia, often studied as a formal thought disorder. The neurobiology of language in schizophrenia has been addressed within the same framework, that language and thought are equivalents considering symptoms and not signs. This review aims to systematically examine published peer-reviewed studies that employed neuroimaging techniques to investigate aberrant brain-language networks in individuals with schizophrenia in relation to linguistic signs.

Methods

We employed a language model for automatic data extraction. We selected our studies according to the PRISMA recommendations, and we conducted the quality assessment of the selected studies according to the STROBE guidance.

Results

We analyzed the findings from 37 studies, categorizing them based on patient characteristics, brain measures, and language task types. The inferior frontal gyrus (IFG) and superior temporal gyrus (STG) exhibited the most significant differences among these studies and paradigms.

Conclusions

We propose guidelines for future research in this field based on our analysis. It is crucial to investigate larger networks involved in language processing, and language models with brain metrics must be integrated to enhance our understanding of the relationship between language and brain abnormalities in schizophrenia.

Keeping track of time: Horizontal spatial biases for hours, days, and months

In many Western cultures, the processing of temporal words related to the past and to the future is associated with left and right space, respectively - a phenomenon known as the horizontal Mental Time Line (MTL). While this mapping is apparently quite ubiquitous, its regularity and consistency across different types of temporal concepts remain to be determined. Moreover, it is unclear whether such spatial mappings are an essential and early constituent of concept activation. In the present study, we used words denoting time units at different scales (hours of the day, days of the week, months of the year) associated with either left space (e.g., 9 a.m., Monday, February) or right space (e.g., 8 p.m., Saturday, November) as cues in a line bisection task. Fifty-seven healthy adults listened to temporal words and then moved a mouse cursor to the perceived midpoint of a horizontally presented line. We measured movement trajectories, initial line intersection coordinates, and final bisection response coordinates. We found movement trajectory displacements for left- vs. right-biasing hour and day cues. Initial line intersections were biased specifically by month cues, while final bisection responses were biased specifically by hour cues. Our findings offer general support to the notion of horizontal space-time associations and suggest further investigation of the exact chronometry and strength of this association across individual time units.

Regionally specific cortical lateralization of abstract and concrete verb processing: Magnetic mismatch negativity study

The neural underpinnings of processing concrete and abstract semantics remain poorly understood. Previous fMRI studies have shown that multimodal and amodal neural networks respond differentially to different semantic types; importantly, abstract semantics activates more left-lateralized networks, as opposed to more bilateral activity for concrete words. Due to the lack of temporal resolution, these fMRI results do not allow to easily separate language- and task-specific brain responses and to disentangle early processing stages from later post-comprehension phenomena. To tackle this, we used magnetoencephalography (MEG), a time-resolved neuroimaging technique, in combination with a task-free oddball mismatch negativity (MMN) paradigm, an established approach to tracking early automatic activation of word-specific memory traces in the brain. We recorded the magnetic MMN responses in 30 healthy adults to auditorily presented abstract and concrete action verbs to assess lateralization of word-specific lexico-semantic processing in a set of neocortical areas. We found that MMN responses to these stimuli showed different lateralization patterns of activity in the upper limb motor area (BA4) and parts of Broca's area (BA45/BA47) within ∼100-350 ms after the word disambiguation point. Importantly, the greater leftward response lateralization for abstract semantics was due to the lesser involvement of the right-hemispheric homologues, not increased left-hemispheric activity. These findings suggest differential region-specific involvement of bilateral sensorimotor systems already in the early automatic stages of processing abstract and concrete action semantics.

Modeling Brain Representations of Words' Concreteness in Context Using GPT-2 and Human Ratings

The meaning of most words in language depends on their context. Understanding how the human brain extracts contextualized meaning, and identifying where in the brain this takes place, remain important scientific challenges. But technological and computational advances in neuroscience and artificial intelligence now provide unprecedented opportunities to study the human brain in action as language is read and understood. Recent contextualized language models seem to be able to capture homonymic meaning variation ("bat", in a baseball vs. a vampire context), as well as more nuanced differences of meaning-for example, polysemous words such as "book", which can be interpreted in distinct but related senses ("explain a book", information, vs. "open a book", object) whose differences are fine-grained. We study these subtle differences in lexical meaning along the concrete/abstract dimension, as they are triggered by verb-noun semantic composition. We analyze functional magnetic resonance imaging (fMRI) activations elicited by Italian verb phrases containing nouns whose interpretation is affected by the verb to different degrees. By using a contextualized language model and human concreteness ratings, we shed light on where in the brain such fine-grained meaning variation takes place and how it is coded. Our results show that phrase concreteness judgments and the contextualized model can predict BOLD activation associated with semantic composition within the language network. Importantly, representations derived from a complex, nonlinear composition process consistently outperform simpler composition approaches. This is compatible with a holistic view of semantic composition in the brain, where semantic representations are modified by the process of composition itself. When looking at individual brain areas, we find that encoding performance is statistically significant, although with differing patterns of results, suggesting differential involvement, in the posterior superior temporal sulcus, inferior frontal gyrus and anterior temporal lobe, and in motor areas previously associated with processing of concreteness/abstractness.

Neural dynamics underlying the processing of implicit form-meaning connections: The dissociative roles of theta and alpha oscillations

Implicit learning plays an important role in the language acquisition. In addition to helping people acquire the form-level rules (e.g., the word order regularities), implicit learning can also facilitate the acquisition of word meanings (i.e., the establishment of connections between the word form and its meanings). Although some behavioral studies have explored the processing of implicit form-meaning connections, the neural dynamics underlying this processing remains unclear. Through examining whether participants could implicitly acquire the literal and metaphorical meanings of novel words, and applying the time-frequency analysis on the electroencephalogram (EEG) data collected in the testing phase, the neural oscillations corresponding to the processing of implicit form-literal and form-metaphorical meaning connections were explored. The results showed that participants in the experimental group could implicitly acquire the form-literal and form-metaphorical meaning connections after training, while participants in the control group who were not trained did not have access to such form-meaning connections. Meanwhile, during the processing of form-literal meaning connections, the greater suppression of alpha oscillations was induced by the testing items that follow the same rules as the training items (i.e., the regular testing items) in the experimental group, whereas the stronger enhancement of theta oscillations was elicited by the regular testing items in the experimental group during the processing of form-metaphorical meaning connections. Our study provides insights for understanding the processing of implicit form-literal and form-metaphorical meaning connections and the neural dynamics underlying the processing.

Embodying Language through Gestures: Residuals of Motor Memories Modulate Motor Cortex Excitability during Abstract Words Comprehension

There is a debate about whether abstract semantics could be represented in a motor domain as concrete language. A contextual association with a motor schema (action or gesture) seems crucial to highlighting the motor system involvement. The present study with transcranial magnetic stimulation aimed to assess motor cortex excitability changes during abstract word comprehension after conditioning word reading to a gesture execution with congruent or incongruent meaning. Twelve healthy volunteers were engaged in a lexical-decision task responding to abstract words or meaningless verbal stimuli. Motor cortex (M1) excitability was measured at different after-stimulus intervals (100, 250, or 500 ms) before and after an associative-learning training where the execution of the gesture followed word processing. Results showed a significant post-training decrease in hand motor evoked potentials at an early processing stage (100 ms) in correspondence to words congruent with the gestures presented during the training. We hypothesized that traces of individual semantic memory, combined with training effects, induced M1 inhibition due to the redundancy of evoked motor representation. No modulation of cortical excitability was found for meaningless or incongruent words. We discuss data considering the possible implications in research to understand the neural basis of language development and language rehabilitation protocols.

Flexing the principal gradient of the cerebral cortex to suit changing semantic task demands

Understanding how thought emerges from the topographical structure of the cerebral cortex is a primary goal of cognitive neuroscience. Recent work has revealed a principal gradient of intrinsic connectivity capturing the separation of sensory-motor cortex from transmodal regions of the default mode network (DMN); this is thought to facilitate memory-guided cognition. However, studies have not explored how this dimension of connectivity changes when conceptual retrieval is controlled to suit the context. We used gradient decomposition of informational connectivity in a semantic association task to establish how the similarity in connectivity across brain regions changes during familiar and more original patterns of retrieval. Multivoxel activation patterns at opposite ends of the principal gradient were more divergent when participants retrieved stronger associations; therefore, when long-term semantic information is sufficient for ongoing cognition, regions supporting heteromodal memory are functionally separated from sensory-motor experience. In contrast, when less related concepts were linked, this dimension of connectivity was reduced in strength as semantic control regions separated from the DMN to generate more flexible and original responses. We also observed fewer dimensions within the neural response towards the apex of the principal gradient when strong associations were retrieved, reflecting less complex or varied neural coding across trials and participants. In this way, the principal gradient explains how semantic cognition is organised in the human cerebral cortex: the separation of DMN from sensory-motor systems is a hallmark of the retrieval of strong conceptual links that are culturally shared.

Effects of social experience on abstract concepts in semantic priming

Humans can understand thousands of abstract words, even when they do not have clearly perceivable referents. Recent views highlight an important role of social experience in grounding of abstract concepts and sub-kinds of abstract concepts, but empirical work in this area is still in its early stages. In the present study, a picture-word semantic priming paradigm was employed to investigate the contribution effect of social experience that is provided by real-life pictures to social abstract (SA, e.g., friendship, betrayal) concepts and emotional abstract (EA, e.g., happiness, anger) concepts. Using a lexical decision task, we examined responses to picture-SA word pairs (Experiment 1) and picture-EA word pairs (Experiment 2) in social/emotional semantically related and unrelated conditions. All pairs shared either positive or negative valence. The results showed quicker responses to positive SA and EA words that were preceded by related vs. unrelated prime pictures. Specifically, positive SA words were facilitated by the corresponding social scene pictures, whereas positive EA words were facilitated by pictures depict the corresponding facial expressions and gestures. However, such facilitatory effect was not observed in negative picture-SA/EA word conditions. This pattern of results suggests that a facilitatory effect of social experience on abstract concepts varies with different sub-kinds of abstract concepts, that seems to be limited to positive SA concepts. Overall, our findings confirm the crucial role of social experience for abstract concepts and further suggest that not all abstract concepts can benefit from social experience, at least in the semantic priming.

Event related potential exploration of the organizational structure of abstract versus concrete words in neurologically intact younger adults

The purpose of this study was to test the effects of concreteness and relationship type (similarity vs. association) on semantic processing using event-related potentials (ERP). Neurophysiological evidence has been found for the concreteness effect and for an effect of relationship type. This study replicated and extended these findings by investigating the interaction of concreteness and relationship type. Twenty-four neurologically healthy young adults performed lexical decision and semantic relatedness tasks while continuous scalp EEG was recorded. Larger N400 effects were found for concrete words in associative relationships than for concrete words in similarity relationships and abstract words in either type of relationship. The results are discussed in relation to the different representational frameworks account for abstract and concrete word processing.

EXPRESS: How negative emotions affect young and older adults' numerosity estimation performance

In this study, we investigated the influence of negative emotions on numerosity estimation and whether this influence changes with aging during adulthood. Young and older adults were asked to estimate and compare the numerosity of collections of elements (cars or dots) with a two-digit number. Collections of elements were preceded by emotionally neutral (e.g., mushrooms) or emotionally negative (e.g., a corpse) pictures. Stimuli were easier (i.e., small-ratio) or harder (i.e., large-ratio) items. Young and older participants obtained similar numerosity estimation performance. Interestingly, participants were less accurate under negative emotions than under neutral emotions when they estimated numerosity of collections of abstract elements (i.e., dots). In contrast, participants improved their performance under negative emotions while estimating collections of non-abstract, daily-life elements (i.e., cars). These findings have important implications for furthering our understanding of the role of negative emotions in numerosity estimation and age-related differences therein.

Evidence for the Concreteness of Abstract Language: A Meta-Analysis of Neuroimaging Studies

The neural mechanisms subserving the processing of abstract concepts remain largely debated. Even within the embodiment theoretical framework, most authors suggest that abstract concepts are coded in a linguistic propositional format, although they do not completely deny the role of sensorimotor and emotional experiences in coding it. To our knowledge, only one recent proposal puts forward that the processing of concrete and abstract concepts relies on the same mechanisms, with the only difference being in the complexity of the underlying experiences. In this paper, we performed a meta-analysis using the Activation Likelihood Estimates (ALE) method on 33 functional neuroimaging studies that considered activations related to abstract and concrete concepts. The results suggest that (1) concrete and abstract concepts share the recruitment of the temporo-fronto-parietal circuits normally involved in the interactions with the physical world, (2) processing concrete concepts recruits fronto-parietal areas better than abstract concepts, and (3) abstract concepts recruit Broca's region more strongly than concrete ones. Based on anatomical and physiological evidence, Broca's region is not only a linguistic region mainly devoted to speech production, but it is endowed with complex motor representations of different biological effectors. Hence, we propose that the stronger recruitment of this region for abstract concepts is expression of the complex sensorimotor experiences underlying it, rather than evidence of a purely linguistic format of its processing.

Neuroscience and climate change: How brain recordings can help us understand human responses to climate change

There is little published neuroscience research on the psychology of climate change. This review outlines how carefully designed experiments that measure key neural processes, linked to specific cognitive processes, can provide powerful tools to answer research questions in climate change psychology. We review relevant literature from social neuroscience that can be applicable to environmental research-the neural correlates of fairness and cooperation, altruistic behaviour and personal values-and discuss important factors when translating environmental psychology constructs to neuroscientific measurement. We provide a practical overview of how to implement environmental neuroscience using electroencephalography, summarising important event-related potential components and how they can be used to answer questions in climate change psychology. Challenges for the field include accurate attribution of findings, both within and between studies, the need for interdisciplinary collaboration, peer review and reporting processes.

Contextual Acquisition of Concrete and Abstract Words: Behavioural and Electrophysiological Evidence

Abstract and concrete words differ in their cognitive and neuronal underpinnings, but the exact mechanisms underlying these distinctions are unclear. We investigated differences between these two semantic types by analysing brain responses to newly learnt words with fully controlled psycholinguistic properties. Experimental participants learned 20 novel abstract and concrete words in the context of short stories. After the learning session, event-related potentials (ERPs) to newly learned items were recorded, and acquisition outcomes were assessed behaviourally in a range of lexical and semantic tasks. Behavioural results showed better performance on newly learnt abstract words in lexical tasks, whereas semantic assessments showed a tendency for higher accuracy for concrete words. ERPs to novel abstract and concrete concepts differed early on, ~150 ms after the word onset. Moreover, differences between novel words and control untrained pseudowords were observed earlier for concrete (~150 ms) than for abstract (~200 ms) words. Distributed source analysis indicated bilateral temporo-parietal activation underpinning newly established memory traces, suggesting a crucial role of Wernicke’s area and its right-hemispheric homologue in word acquisition. In sum, we report behavioural and neurophysiological processing differences between concrete and abstract words evident immediately after their controlled acquisition, confirming distinct neurocognitive mechanisms underpinning these types of semantics.

Psychological and Electrophysiological Correlates of Word Learning Success

Background

A rich vocabulary supports human achievements in socio-economic activities, education, and communication. It is therefore important to clarify the nature of language acquisition as a complex multidimensional process. However, both the psychological and neurophysiological mechanisms underpinning language learning, as well as the links between them, are still poorly understood.

Objective

This study aims to explore the psychological and neurophysiological correlates of successful word acquisition in a person's native language.

Design

Thirty adults read sentences with novel nouns, following which the participants' electroencephalograms were recorded during a word-reading task. Event- related potentials in response to novel words and alpha oscillation parameters (amplitude, variability, and long-range temporal correlation dynamics) were analyzed. Learning outcomes were assessed at the lexical and semantic levels. Psychological variables measured using Amthauer's test (verbal abilities), BIS/BAS scales (motivation), and the MSTAT-1 (ambiguity tolerance) and alpha oscillation parameters were factored.

Results

Better recognition of novel words was related to two factors which had high factor loadings for all measured alpha oscillation parameters, indicating the role of attention networks and respective neural activity for enabling information processing. More successful learners had lower P200 amplitude, which also suggests higher attention-system involvement. Another factor predicted better acquisition of word meanings for less ambiguity-tolerant students, while the factor which pooled logical conceptual thinking ability and persistence in goal-reaching, positively correlated with acquisition of both word forms and meanings.

Conclusion

The psychological factors predominantly correlated with word-learning success in semantic tasks, while neurophysiological variables were linked to performance in the recognition task.

Corroborating behavioral evidence for the interplay of representational richness and semantic control in semantic word processing

This study aimed to replicate and validate concreteness and context effects on semantic word processing. In Experiment 1, we replicated the behavioral findings of Hoffman et al. (Cortex 63,250–266, https://doi.org/10.1016/j.cortex.2014.09.001, 2015) by applying their cueing paradigm with their original stimuli translated into German. We found concreteness and contextual cues to facilitate word processing in a semantic judgment task with 55 healthy adults. The two factors interacted in their effect on reaction times: abstract word processing profited more strongly from a contextual cue, while the concrete words’ processing advantage was reduced but still present. For accuracy, the descriptive pattern of results suggested an interaction, which was, however, not significant. In Experiment 2, we reformulated the contextual cues to avoid repetition of the to-be-processed word. In 83 healthy adults, the same pattern of results emerged, further validating the findings. Our corroborating evidence supports theories integrating representational richness and semantic control mechanisms as complementary mechanisms in semantic word processing.

Expert Programmers Have Fine-Tuned Cortical Representations of Source Code

Expertise enables humans to achieve outstanding performance on domain-specific tasks, and programming is no exception. Many studies have shown that expert programmers exhibit remarkable differences from novices in behavioral performance, knowledge structure, and selective attention. However, the underlying differences in the brain of programmers are still unclear. We here address this issue by associating the cortical representation of source code with individual programming expertise using a data-driven decoding approach. This approach enabled us to identify seven brain regions, widely distributed in the frontal, parietal, and temporal cortices, that have a tight relationship with programming expertise. In these brain regions, functional categories of source code could be decoded from brain activity and the decoding accuracies were significantly correlated with individual behavioral performances on a source-code categorization task. Our results suggest that programming expertise is built on fine-tuned cortical representations specialized for the domain of programming.

Acquisition of concrete and abstract words is modulated by tDCS of Wernicke's area

Previous behavioural and neuroimaging research suggested distinct cortical systems involved in processing abstract and concrete semantics; however, there is a dearth of causal evidence to support this. To address this, we applied anodal, cathodal, or sham (placebo) tDCS over Wernicke’s area before a session of contextual learning of novel concrete and abstract words (n = 10 each), presented five times in short stories. Learning effects were assessed at lexical and semantic levels immediately after the training and, to attest any consolidation effects of overnight sleep, on the next day. We observed successful learning of all items immediately after the session, with decreased performance in Day 2 assessment. Importantly, the results differed between stimulation conditions and tasks. Whereas the accuracy of semantic judgement for abstract words was significantly lower in the sham and anodal groups on Day 2 vs. Day 1, no significant performance drop was observed in the cathodal group. Similarly, the cathodal group showed no significant overnight performance reduction in the free recall task for either of the stimuli, unlike the other two groups. Furthermore, between-group analysis showed an overall better performance of both tDCS groups over the sham group, particularly expressed for abstract semantics and cathodal stimulation. In sum, the results suggest overlapping but diverging brain mechanisms for concrete and abstract semantics and indicate a larger degree of involvement of core language areas in storing abstract knowledge. Furthermore, they demonstrate a possiblity to improve learning outcomes using neuromodulatory techniques.

Dual-task performance of speech and motor skill: verb generation facilitates grasping behaviour

Pronouncing nouns or verbs while grasping distinctly alters movement. Changes in hand speed and final position occur according to the meaning of the words spoken. These results are typically found when executing a single movement paired with a single word. For example, pronouncing the word 'fast' increased the speed of the hand when reaching to grasp. Our objective was to compare how verb and noun fluency tasks interact with grasping behaviour in a grasp-to-construct task. Because previous imaging research shows that verb and noun production activates distinct neural areas, we reasoned that grasping outcomes would differ according to the category of word produced by participants. Specifically, we hypothesized that verb pronunciation would distinctly affect grasping behaviour compared to producing nouns. We recruited 38 young adults who performed a grasp-to-construct task and two different verbal fluency tasks. Participants completed each task (grasp, verb fluency, and noun fluency) separately as control conditions, and the grasping and each speaking task simultaneously for dual-task conditions. We found that during the dual-task condition, when generating nouns and grasping, participants made significantly more grasping errors (inaccurate grasps) compared to the control and verb dual-task conditions. Moreover, our results revealed a relationship between the number of verbs generated and grasping performance. Participants who generated more verbs were faster and more accurate during the motor component of the dual-task condition. This relationship was not observed when nouns were produced, indicating a unique relationship between verb production and functional grasping. The result is a facilitation effect, diminishing the negative outcome on motor control associated with increased cognitive load (as observed during noun pronunciation).