On the mental representations originating during the interaction between language and vision

Ultimate Grounding of Abstract Concepts: A Graded Account

Abstraction, one of the hallmarks of human cognition, continues to be the topic of a strong debate. The primary disagreement concerns whether or not abstract concepts can be accounted for within the scope of embodied cognition. In this paper, we introduce the embodied approach to conceptual knowledge and distinguish between embodiment and grounding, where grounding is the general term for how concepts initially acquire their meaning. Referring to numerous pieces of empirical evidence, we emphasise that, ultimately, all concepts are acquired via interaction with the world via two main pathways: embodiment and social interaction. The first pathway is direct and primarily involves action/perception, interoception and emotions. The second pathway is indirect, being mediated by language in particular. Evidence from neuroscience, psychology and cognitive linguistics shows these pathways have different properties, roles in cognition and temporal profiles. Human development also places revealing constraints on how children develop the ability to reason more abstractly as they grow up. We recognize language as a crucial cognitive faculty with several roles enabling the acquisition of abstract concepts indirectly. Three detailed case studies on body-specificity hypothesis, abstract verbs and mathematics are used to argue that a compelling case has accumulated in favour of the ultimate grounding of abstract concepts in an agent’s interaction with its world, primarily relying on the direct pathway. We consolidate the debate through multidisciplinary evidence for the idea that abstractness is a graded, rather than a binary property of concepts.

Attention to body-parts varies with visual preference and verb-effector associations

Theories of embodied conceptual meaning suggest fundamental relations between others' actions, language, and our own actions and visual attention processes. Prior studies have found that when people view an image of a neutral body in a scene they first look toward, in order, the head, torso, hands, and legs. Other studies show associations between action verbs and the body-effectors used in performing the action (e.g., "jump" with feet/legs; "talk" with face/head). In the present experiment, the visual attention of participants was recorded with a remote eye-tracking system while they viewed an image of an actor pantomiming an action and heard a concrete action verb. Participants manually responded whether or not the action image was a good example of the verb they heard. The eye-tracking results confirmed that participants looked at the head most, followed by the hands, and the feet least of all; however, visual attention to each of the body-parts also varied as a function of the effector associated with the spoken verb on image/verb congruent trials, particularly for verbs associated with the legs. Overall, these results suggest that language influences some perceptual processes; however, hearing auditory verbs did not alter the previously reported fundamental hierarchical sequence of directed attention, and fixations on specific body-effectors may not be essential for verb comprehension as peripheral visual cues may be sufficient to perform the task.

The Influence of Motor Imagery on Postural Sway: Differential Effects of Type of Body Movement and Person Perspective

The present study examined the differential effects of kinesthetic imagery (first person perspective) and visual imagery (third person perspective) on postural sway during quiet standing. Based on an embodied cognition perspective, the authors predicted that kinesthetic imagery would lead to activations in movement-relevant motor systems to a greater degree than visual imagery. This prediction was tested among 30 participants who imagined various motor activities from different visual perspectives while standing on a strain gauge plate. The results showed that kinesthetic imagery of lower body movements, but not of upper body movements, had clear effects on postural parameters (sway path length and frequency contents of sway). Visual imagery, in contrast, had no reliable effects on postural activity. We also found that postural effects were not affected by the vividness of imagery. The results suggest that during kinesthetic motor imagery participants partially simulated (re-activated) the imagined movements, leading to unintentional postural adjustments. These findings are consistent with an embodied cognition perspective on motor imagery.

Neural correlates of visualizations of concrete and abstract words in preschool children: a developmental embodied approach

The neural correlates of visualization underlying word comprehension were examined in preschool children. On each trial, a concrete or abstract word was delivered binaurally (part 1: post-auditory visualization), followed by a four-picture array (a target plus three distractors; part 2: matching visualization). Children were to select the picture matching the word they heard in part 1. Event-related potentials (ERPs) locked to each stimulus presentation and task interval were averaged over sets of trials of increasing word abstractness. ERP time-course during both parts of the task showed that early activity (i.e., <300 ms) was predominant in response to concrete words, while activity in response to abstract words became evident only at intermediate (i.e., 300–699 ms) and late (i.e., 700–1000 ms) ERP intervals. Specifically, ERP topography showed that while early activity during post-auditory visualization was linked to left temporo-parietal areas for concrete words, early activity during matching visualization occurred mostly in occipito-parietal areas for concrete words, but more anteriorly in centro-parietal areas for abstract words. In intermediate ERPs, post-auditory visualization coincided with parieto-occipital and parieto-frontal activity in response to both concrete and abstract words, while in matching visualization a parieto-central activity was common to both types of words. In the late ERPs for both types of words, the post-auditory visualization involved right-hemispheric activity following a “post-anterior” pathway sequence: occipital, parietal, and temporal areas; conversely, matching visualization involved left-hemispheric activity following an “ant-posterior” pathway sequence: frontal, temporal, parietal, and occipital areas. These results suggest that, similarly, for concrete and abstract words, meaning in young children depends on variably complex visualization processes integrating visuo-auditory experiences and supramodal embodying representations.

Event-related potential signatures of perceived and imagined emotional and food real-life photos

Although food and affective pictures share similar emotional and motivational characteristics, the relationship between the neuronal responses to these stimuli is unclear. Particularly, it is not known whether perceiving and imagining food and affective stimuli elicit similar event-related potential (ERP) patterns. In this study, two ERP correlates, the early posterior negativity (EPN) and the late positive potential (LPP) for perceived and imagined emotional and food photographs were investigated. Thirteen healthy volunteers were exposed to a set of food photos, as well as unpleasant, pleasant, and neutral photos from the International Affective Picture System. In each trial, participants were first asked to view a photo (perception condition), and then to create a visual mental image of it and to rate its vividness (imagery condition). The results showed that during perception, brain regions corresponding to sensorimotor and parietal motivational (defensive and appetitive) systems were activated to different extents, producing a graded pattern of EPN and LPP responses specific to the photo content - more prominent for unpleasant than pleasant and food content. Also, an EPN signature occurred in both conditions for unpleasant content, suggesting that, compared to food or pleasant content, unpleasant content may be attended to more intensely during perception and may be represented more distinctly during imagery. Finally, compared to LLP activation during perception, as well as imagery and perception of all other content, LPP activation was significantly reduced during imagery of unpleasant photos, suggesting inhibition of unwanted memories. Results are framed within a neurocognitive working model of embodied emotions.

The linguistic context effects on the processing of body-object interaction words: An ERP study on second language learners

Embodied theories of cognition argue that the processing of both concrete and abstract concepts requires the activation of sensorimotor systems. The present study examined the time course for embedding a sensorimotor context in order to elicit sensitivity to the sensorimotor consequences of understanding body-object interaction (BOI) words. In the study, Event-Related Potentials (ERPs) were recorded while subjects performed a sentence acceptability task. Target BOI words were preceded by rich or poor sensorimotor sentential contexts. The behavioural results replicated previous findings in that high BOI words received a response faster than low BOI words. In addition to this, however, there was a context effect in the sensorimotor region as well as a BOI effect in the parietal region (involved in object representation). The results indicate that the sentential sensorimotor context contributes to the subsequent BOI processing and that action-and perception-related language leads to the activation of the same brain areas, which is consistent with the embodiment theory.

Eye movements reveal the dynamic simulation of speed in language

This study investigates how speed of motion is processed in language. In three eye-tracking experiments, participants were presented with visual scenes and spoken sentences describing fast or slow events (e.g., The lion ambled/dashed to the balloon). Results showed that looking time to relevant objects in the visual scene was affected by the speed of verb of the sentence, speaking rate, and configuration of a supporting visual scene. The results provide novel evidence for the mental simulation of speed in language and show that internal dynamic simulations can be played out via eye movements toward a static visual scene.

Appraisal of space words and allocation of emotion words in bodily space

The body-specificity hypothesis (BSH) predicts that right-handers and left-handers allocate positive and negative concepts differently on the horizontal plane, i.e., while left-handers allocate negative concepts on the right-hand side of their bodily space, right-handers allocate such concepts to the left-hand side. Similar research shows that people, in general, tend to allocate positive and negative concepts in upper and lower areas, respectively, in relation to the vertical plane. Further research shows a higher salience of the vertical plane over the horizontal plane in the performance of sensorimotor tasks. The aim of the paper is to examine whether there should be a dominance of the vertical plane over the horizontal plane, not only at a sensorimotor level but also at a conceptual level. In Experiment 1, various participants from diverse linguistic backgrounds were asked to rate the words "up", "down", "left", and "right". In Experiment 2, right-handed participants from two linguistic backgrounds were asked to allocate emotion words into a square grid divided into four boxes of equal areas. Results suggest that the vertical plane is more salient than the horizontal plane regarding the allocation of emotion words and positively-valenced words were placed in upper locations whereas negatively-valenced words were placed in lower locations. Together, the results lend support to the BSH while also suggesting a higher saliency of the vertical plane over the horizontal plane in the allocation of valenced words.

When the body is time: spatial and temporal deixis in children with visual impairments and sighted children

While there is mounting evidence explaining how concrete concepts are processed, the evidence demonstrating how abstract concepts are processed is rather scant. Most research illustrating how concrete and abstract concepts are processed has been obtained from adult populations. Consequently, not much is known about how these concepts are processed by children, especially those with sensorimotor impairments. This paper reports a study in which groups of children who were either visual-motor impaired (VMG), blind (BG), or sighted (CG) were requested to perform deictic gestures for temporal and spatial concepts. The results showed that: (i) spatial pointing was performed faster than temporal pointing across all groups of children; (ii) such difference in pointing times occurred also within groups; and (iii) the slowest pointing times were those of the blind children followed by the VMG and the CG children, respectively. Additionally, while CG children correctly performed the pointing tasks, VMG and, particularly, BG children relied on a form of deixis known as autotopological (or personal) deixis. The results thus suggest that deprivation or lack of sensorimotor experience with the environment affects the processing of abstract concepts and that a compensatory mechanism may be to rely on the body as a reference frame.

Mechanisms and representations of language-mediated visual attention

The experimental investigation of language-mediated visual attention is a promising way to study the interaction of the cognitive systems involved in language, vision, attention, and memory. Here we highlight four challenges for a mechanistic account of this oculomotor behavior: the levels of representation at which language-derived and vision-derived representations are integrated; attentional mechanisms; types of memory; and the degree of individual and group differences. Central points in our discussion are (a) the possibility that local microcircuitries involving feedforward and feedback loops instantiate a common representational substrate of linguistic and non-linguistic information and attention; and (b) that an explicit working memory may be central to explaining interactions between language and visual attention. We conclude that a synthesis of further experimental evidence from a variety of fields of inquiry and the testing of distinct, non-student, participant populations will prove to be critical.