Exploring the Neural Representation of Novel Words Learned through Enactment in a Word Recognition Task

Brain regions supporting retrieval of words drawn at encoding: fMRI evidence for multimodal reactivation

Memory for words that are drawn or sketched by the participant, rather than written, during encoding is typically superior. While this drawing benefit has been reliably demonstrated in recent years, there has yet to be an investigation of its neural basis. Here, we asked participants to either create drawings, repeatedly write, or list physical characteristics depicting each target word during encoding. Participants then completed a recognition memory test for target words while undergoing functional magnetic resonance imaging (fMRI). Behavioural results showed memory was significantly higher for words drawn than written, replicating the typical drawing effect. Memory for words whose physical characteristics were listed at encoding was also higher than for those written repeatedly, but lower than for those drawn. Voxel-wise analyses of fMRI data revealed two distributed sets of brain regions more active for items drawn relative to written, the left angular gyrus (BA 39) and bilateral frontal (BA 10) regions, suggesting integration and self-referential processing during retrieval of drawn words. Brain-behaviour correlation analyses showed that the size of one's memory benefit for words drawn relative to written at encoding was positively correlated with activation in brain regions linked to visual representation and imagery (BA 17 and cuneus) and motor planning (premotor and supplementary motor areas; BA 6). This study suggests that drawing benefits memory by coactivating multiple sensory traces. Target words drawn during encoding are subsequently remembered by re-engaging visual, motoric, and semantic representations.

Gestures as Scaffolding to Learn Vocabulary in a Foreign Language

This paper investigates the influence of gestures on foreign language (FL) vocabulary learning. In this work, we first address the state of the art in the field and then delve into the research conducted in our lab (three experiments already published) in order to finally offer a unified theoretical interpretation of the role of gestures in FL vocabulary learning. In Experiments 1 and 2, we examined the impact of gestures on noun and verb learning. The results revealed that participants exhibited better learning outcomes when FL words were accompanied by congruent gestures compared to those from the no-gesture condition. Conversely, when meaningless or incongruent gestures were presented alongside new FL words, gestures had a detrimental effect on the learning process. Secondly, we addressed the question of whether or not individuals need to physically perform the gestures themselves to observe the effects of gestures on vocabulary learning (Experiment 3). Results indicated that congruent gestures improved FL word recall when learners only observed the instructor's gestures ("see" group) and when they mimicked them ("do" group). Importantly, the adverse effect associated with incongruent gestures was reduced in the "do" compared to that in the "see" experimental group. These findings suggest that iconic gestures can serve as an effective tool for learning vocabulary in an FL, particularly when the gestures align with the meaning of the words. Furthermore, the active performance of gestures helps counteract the negative effects associated with inconsistencies between gestures and word meanings. Consequently, if a choice must be made, an FL learning strategy in which learners acquire words while making gestures congruent with their meaning would be highly desirable.

Benefits of Enacting and Observing Gestures on Foreign Language Vocabulary Learning: A Systematic Review and Meta-Analysis

The integration of physical movements, such as gestures, into learning holds potential for enhancing foreign language (L2) education. Uncovering whether actively performing gestures during L2 learning is more, or equally, effective compared to simply observing such movements is central to deepening our understanding of the efficacy of movement-based learning strategies. Here, we present a meta-analysis of seven studies containing 309 participants that compares the effects of gesture self-enactment and observation on L2 vocabulary learning. The results showed that gesture observation was just as effective for L2 learning as gesture enactment, based on free recall, cued L2 recognition, and cued native language recognition performance, with a large dispersion of true effect across studies. Gesture observation may be sufficient for inducing embodied L2 learning benefits, in support of theories positing shared mechanisms underlying enactment and observation. Future studies should examine the effects of gesture-based learning over longer time periods with larger sample sizes and more diverse word classes.

Enriched learning: behavior, brain, and computation

The presence of complementary information across multiple sensory or motor modalities during learning, referred to as multimodal enrichment, can markedly benefit learning outcomes. Why is this? Here, we integrate cognitive, neuroscientific, and computational approaches to understanding the effectiveness of enrichment and discuss recent neuroscience findings indicating that crossmodal responses in sensory and motor brain regions causally contribute to the behavioral benefits of enrichment. The findings provide novel evidence for multimodal theories of enriched learning, challenge assumptions of longstanding cognitive theories, and provide counterevidence to unimodal neurobiologically inspired theories. Enriched educational methods are likely effective not only because they may engage greater levels of attention or deeper levels of processing, but also because multimodal interactions in the brain can enhance learning and memory.

Embodying Language through Gestures: Residuals of Motor Memories Modulate Motor Cortex Excitability during Abstract Words Comprehension

There is a debate about whether abstract semantics could be represented in a motor domain as concrete language. A contextual association with a motor schema (action or gesture) seems crucial to highlighting the motor system involvement. The present study with transcranial magnetic stimulation aimed to assess motor cortex excitability changes during abstract word comprehension after conditioning word reading to a gesture execution with congruent or incongruent meaning. Twelve healthy volunteers were engaged in a lexical-decision task responding to abstract words or meaningless verbal stimuli. Motor cortex (M1) excitability was measured at different after-stimulus intervals (100, 250, or 500 ms) before and after an associative-learning training where the execution of the gesture followed word processing. Results showed a significant post-training decrease in hand motor evoked potentials at an early processing stage (100 ms) in correspondence to words congruent with the gestures presented during the training. We hypothesized that traces of individual semantic memory, combined with training effects, induced M1 inhibition due to the redundancy of evoked motor representation. No modulation of cortical excitability was found for meaningless or incongruent words. We discuss data considering the possible implications in research to understand the neural basis of language development and language rehabilitation protocols.

Twelve- and Fourteen-Year-Old School Children Differentially Benefit from Sensorimotor- and Multisensory-Enriched Vocabulary Training

Both children and adults have been shown to benefit from the integration of multisensory and sensorimotor enrichment into pedagogy. For example, integrating pictures or gestures into foreign language (L2) vocabulary learning can improve learning outcomes relative to unisensory learning. However, whereas adults seem to benefit to a greater extent from sensorimotor enrichment such as the performance of gestures in contrast to multisensory enrichment with pictures, this is not the case in elementary school children. Here, we compared multisensory- and sensorimotor-enriched learning in an intermediate age group that falls between the age groups tested in previous studies (elementary school children and young adults), in an attempt to determine the developmental time point at which children’s responses to enrichment mature from a child-like pattern into an adult-like pattern. Twelve-year-old and fourteen-year-old German children were trained over 5 consecutive days on auditorily presented, concrete and abstract, Spanish vocabulary. The vocabulary was learned under picture-enriched, gesture-enriched, and non-enriched (auditory-only) conditions. The children performed vocabulary recall and translation tests at 3 days, 2 months, and 6 months post-learning. Both picture and gesture enrichment interventions were found to benefit children’s L2 learning relative to non-enriched learning up to 6 months post-training. Interestingly, gesture-enriched learning was even more beneficial than picture-enriched learning for the 14-year-olds, while the 12-year-olds benefitted equivalently from learning enriched with pictures and gestures. These findings provide evidence for opting to integrate gestures rather than pictures into L2 pedagogy starting at 14 years of age.

Motor Cortex Causally Contributes to Vocabulary Translation following Sensorimotor-Enriched Training

The role of the motor cortex in perceptual and cognitive functions is highly controversial. Here, we investigated the hypothesis that the motor cortex can be instrumental for translating foreign language vocabulary. Human participants of both sexes were trained on foreign language (L2) words and their native language translations over 4 consecutive days. L2 words were accompanied by complementary gestures (sensorimotor enrichment) or pictures (sensory enrichment). Following training, participants translated the auditorily presented L2 words that they had learned. During translation, repetitive transcranial magnetic stimulation was applied bilaterally to a site within the primary motor cortex (Brodmann area 4) located in the vicinity of the arm functional compartment. Responses within the stimulated motor region have previously been found to correlate with behavioral benefits of sensorimotor-enriched L2 vocabulary learning. Compared to sham stimulation, effective perturbation by repetitive transcranial magnetic stimulation slowed down the translation of sensorimotor-enriched L2 words, but not sensory-enriched L2 words. This finding suggests that sensorimotor-enriched training induced changes in L2 representations within the motor cortex, which in turn facilitated the translation of L2 words. The motor cortex may play a causal role in precipitating sensorimotor-based learning benefits, and may directly aid in remembering the native language translations of foreign language words following sensorimotor-enriched training. These findings support multisensory theories of learning while challenging reactivation-based theories.SIGNIFICANCE STATEMENT Despite the potential for sensorimotor enrichment to serve as a powerful tool for learning in many domains, its underlying brain mechanisms remain largely unexplored. Using transcranial magnetic stimulation and a foreign language (L2) learning paradigm, we found that sensorimotor-enriched training can induce changes in L2 representations within the motor cortex, which in turn causally facilitate the translation of L2 words. The translation of recently acquired L2 words may therefore rely not only on auditory information stored in memory or on modality-independent L2 representations, but also on the sensorimotor context in which the words have been experienced.

Visual recognition of words learned with gestures induces motor resonance in the forearm muscles

According to theories of Embodied Cognition, memory for words is related to sensorimotor experiences collected during learning. At a neural level, words encoded with self-performed gestures are represented in distributed sensorimotor networks that resonate during word recognition. Here, we ask whether muscles involved in gesture execution also resonate during word recognition. Native German speakers encoded words by reading them (baseline condition) or by reading them in tandem with picture observation, gesture observation, or gesture observation and execution. Surface electromyogram (EMG) activity from both arms was recorded during the word recognition task and responses were detected using eye-tracking. The recognition of words encoded with self-performed gestures coincided with an increase in arm muscle EMG activity compared to the recognition of words learned under other conditions. This finding suggests that sensorimotor networks resonate into the periphery and provides new evidence for a strongly embodied view of recognition memory.

Visual Sensory Cortices Causally Contribute to Auditory Word Recognition Following Sensorimotor-Enriched Vocabulary Training

Despite a rise in the use of "learning by doing" pedagogical methods in praxis, little is known as to how the brain benefits from these methods. Learning by doing strategies that utilize complementary information ("enrichment") such as gestures have been shown to optimize learning outcomes in several domains including foreign language (L2) training. Here we tested the hypothesis that behavioral benefits of gesture-based enrichment are critically supported by integrity of the biological motion visual cortices (bmSTS). Prior functional neuroimaging work has implicated the visual motion cortices in L2 translation following sensorimotor-enriched training; the current study is the first to investigate the causal relevance of these structures in learning by doing contexts. Using neuronavigated transcranial magnetic stimulation and a gesture-enriched L2 vocabulary learning paradigm, we found that the bmSTS causally contributed to behavioral benefits of gesture-enriched learning. Visual motion cortex integrity benefitted both short- and long-term learning outcomes, as well as the learning of concrete and abstract words. These results adjudicate between opposing predictions of two neuroscientific learning theories: While reactivation-based theories predict no functional role of specialized sensory cortices in vocabulary learning outcomes, the current study supports the predictive coding theory view that these cortices precipitate sensorimotor-based learning benefits.

Absence of post-learning motor activity effects on memory for motor-related words

Drawing a hypothesis from embodied theories of memory, van Dam, Rueschemeyer, Bekkering and Lindemann [(2013). Embodied grounding of memory: Toward the effects of motor execution on memory consolidation. The Quarterly Journal of Experimental Psychology, 66(12), 2310-2328] showed that recognition performance for action words could be modulated by actions performed during the retention interval, suggesting that motor actions during the retention interval affect memory consolidation. The results of 4 experiments from two different laboratories, designed to replicate and extend the van Dam et al. motor consolidation effect, are presented here. Two of the experiments (n = 30 and n = 44) exactly and independently replicated the experimental design and conditions of the original experiment. Yes/No recognition scores plus additional analysis of response times showed no motor consolidation effects. A third experiment (n = 44) manipulating type of processing during encoding also failed to find significant motor consolidation effects. Finally, a fourth experiment (n = 120) following a more standard reconsolidation paradigm, involving 24-hour intervals between learning and motor behaviour, and a 24-hour delayed test, also found null effects. The absence of effects of motor execution on memory consolidation is discussed in terms of the implications of these findings for the embodiment approach to cognition.

How words ripple through bilingual hands: Motor-language coupling during L1 and L2 writing

The speed of our hand movements can be affected by concurrent processing of manual action verbs (MaVs). Whereas this phenomenon is well established for native languages (L1s), it remains underexplored in late foreign languages (L2s), especially during highly automatized tasks. Here we timed keystroke activity while Spanish-English bilinguals copied MaVs, non-manual action verbs, and non-action verbs in their L1 and L2. Motor planning and execution dynamics were indexed by first-letter lag (the time-lapse between word presentation and first keystroke) and whole-word lag (the time-lapse between first and last keystroke), respectively. Despite yielding no effects on motor planning, MaVs facilitated typing execution in L1 but delayed it in L2, irrespective of the subjects' typing skills, age of L2 learning, and L2 competence. Therefore, motor-language coupling effects seem to be present in both languages though they can arise differently in each. These results extend language grounding models, illuminating the role of embodied mechanisms throughout life.

Positive effects of grasping virtual objects on memory for novel words in a second language

Theories of embodied cognition describe language processing and representation as inherently connected to the sensorimotor experiences collected during acquisition. While children grasp their world, collect bodily experiences and name them, in second language (L2), students learn bilingual word lists. Experimental evidence shows that embodiment by mean of gestures enhances memory for words in L2. However, no study has been conducted on the effects of grasping in L2. In a virtual scenario, we trained 46 participants on 18 two- and three-syllabic words of Vimmi, an artificial corpus created for experimental purposes. The words were assigned concrete meanings of graspable objects. Six words were learned audio-visually, by reading the words projected on the wall and by hearing them. Another 6 words were trained by observation of virtual objects. Another 6 words were learned by observation and additional grasping the virtual objects. Thereafter participants were subministered free, cued recall, and reaction time tests in order to assess the word retention and the word recognition. After 30 days, the recall tests were repeated remotely to assess the memory in the long term. The results show that grasping of virtual objects can lead to superior memory performance and to lower reaction times during recognition.

Learning Foreign Language Vocabulary with Gestures and Pictures Enhances Vocabulary Memory for Several Months Post-Learning in Eight-Year-Old School Children

The integration of gestures and pictures into pedagogy has demonstrated potential for improving adults’ learning of foreign language (L2) vocabulary. However, the relative benefits of gestures and pictures on children’s L2 vocabulary learning have not been formally evaluated. In three experiments, we investigated the effects of gesture-based and picture-based learning on 8-year-old primary school children’s acquisition of novel L2 vocabulary. In each experiment, German children were trained over 5 consecutive days on auditorily presented, concrete and abstract, English vocabulary. In Experiments 1 and 2, gesture enrichment (auditorily presented L2 words accompanied with self-performed gestures) was compared with a non-enriched baseline condition. In Experiment 3, gesture enrichment was compared with picture enrichment (auditorily presented words accompanied with pictures). Children performed vocabulary recall and translation tests at 3 days, 2 months, and 6 months post-learning. Both gesture and picture enrichment enhanced children’s test performance compared with non-enriched learning. Benefits of gesture and picture enrichment persisted up to 6 months after training and occurred for both concrete and abstract words. Gesture-enriched learning was hypothesized to boost learning outcomes more than picture-enriched learning on the basis of previous findings in adults. Unexpectedly, however, we observed similar benefits of gesture and picture enrichment on children’s L2 learning. These findings suggest that both gestures and pictures enhance children’s L2 learning and that performance benefits are robust over long timescales.

Too late to be grounded? Motor resonance for action words acquired after middle childhood

Though well established for languages acquired in infancy, the role of embodied mechanisms remains poorly understood for languages learned in middle childhood and adulthood. To bridge this gap, we examined 34 experiments that assessed sensorimotor resonance during processing of action-related words in real and artificial languages acquired since age 7 and into adulthood. Evidence from late bilinguals indicates that foreign-language action words modulate neural activity in motor circuits and predictably facilitate or delay physical movements (even in an effector-specific fashion), with outcomes that prove partly sensitive to language proficiency. Also, data from newly learned vocabularies suggest that embodied effects emerge after brief periods of adult language exposure, remain stable through time, and hinge on the performance of bodily movements (and, seemingly, on action observation, too). In sum, our work shows that infant language exposure is not indispensable for the recruitment of embodied mechanisms during language processing, a finding that carries non-trivial theoretical, pedagogical, and clinical implications for neurolinguistics, in general, and bilingualism research, in particular.

Embodied Learning: Why at School the Mind Needs the Body

Despite all methodological efforts made in the last three decades, Western instruction grounds on traditional principles. Most educational programs follow theories that are mentalistic, i.e., they separate the mind from the body. At school, learners sit, watch, listen, and write. The aim of this paper is to present embodied learning as an alternative to mentalistic education. Similarly, this paper wants to describe embodied learning from a neuroscientific perspective. After a brief historical overview, I will review studies highlighting the behavioral effectiveness of embodied instruction in second language learning, mathematics and spatial thinking. On this base, I will discuss some of the brain mechanisms driving embodied learning and describe its advantages, clearly pleading in favor of instructional practice that reunites body and mind.

Auditory-motor coupling affects phonetic encoding

Recent studies have shown that moving in synchrony with auditory stimuli boosts attention allocation and verbal learning. Furthermore rhythmic tones are processed more efficiently than temporally random tones ('timing effect'), and this effect is increased when participants actively synchronize their motor performance with the rhythm of the tones, resulting in auditory-motor synchronization. Here, we investigated whether this applies also to sequences of linguistic stimuli (syllables). We compared temporally irregular syllable sequences with two temporally regular conditions where either the interval between syllable onsets (stimulus onset asynchrony, SOA) or the interval between the syllables' vowel onsets was kept constant. Entrainment to the stimulus presentation frequency (1 Hz) and event-related potentials were assessed in 24 adults who were instructed to detect pre-defined deviant syllables while they either pedaled or sat still on a stationary exercise bike. We found larger 1 Hz entrainment and P300 amplitudes for the SOA presentation during motor activity. Furthermore, the magnitude of the P300 component correlated with the motor variability in the SOA condition and 1 Hz entrainment, while in turn 1 Hz entrainment correlated with auditory-motor synchronization performance. These findings demonstrate that acute auditory-motor coupling facilitates phonetic encoding.

Embodied Semantics in a Second Language: Critical Review and Clinical Implications

The role of the sensorimotor system in second language (L2) semantic processing as well as its clinical implications for bilingual patients has hitherto been neglected. We offer an overview of the issues at stake in this under-investigated field, presenting the theoretical and clinical relevance of studying L2 embodiment and reviewing the few studies on this topic. We highlight that (a) the sensorimotor network is involved in L2 processing, and that (b) in most studies, L2 is differently embodied than L1, reflected in a lower degree or in a different pattern of L2 embodiment. Importantly, we outline critical issues to be addressed in order to guide future research. We also delineate the subsequent steps needed to confirm or dismiss the value of language therapeutic approaches based on embodiment theories as a complement of speech and language therapies in adult bilinguals.

Depth of Encoding Through Observed Gestures in Foreign Language Word Learning

Word learning is basic to foreign language acquisition, however time consuming and not always successful. Empirical studies have shown that traditional (visual) word learning can be enhanced by gestures. The gesture benefit has been attributed to depth of encoding. Gestures can lead to depth of encoding because they trigger semantic processing and sensorimotor enrichment of the novel word. However, the neural underpinning of depth of encoding is still unclear. Here, we combined an fMRI and a behavioral study to investigate word encoding online. In the scanner, participants encoded 30 novel words of an artificial language created for experimental purposes and their translation into the subjects' native language. Participants encoded the words three times: visually, audiovisually, and by additionally observing semantically related gestures performed by an actress. Hemodynamic activity during word encoding revealed the recruitment of cortical areas involved in stimulus processing. In this study, depth of encoding can be spelt out in terms of sensorimotor brain networks that grow larger the more sensory modalities are linked to the novel word. Word retention outside the scanner documented a positive effect of gestures in a free recall test in the short term.

Guided Embodiment and Potential Applications of Tutor Systems in Language Instruction and Rehabilitation

Intelligent tutor systems (ITSs) in mobile devices take us through learning tasks and make learning ubiquitous, autonomous, and at low cost (Nye, 2015). In this paper, we describe guided embodiment as an ITS essential feature for second language learning (L2) and aphasia rehabilitation (ARe) that enhances efficiency in the learning process. In embodiment, cognitive processes, here specifically language (re)learning are grounded in actions and gestures (Pecher and Zwaan, 2005; Fischer and Zwaan, 2008; Dijkstra and Post, 2015). In order to guide users through embodiment, ITSs must track action and gesture, and give corrective feed-back to achieve the users' goals. Therefore, sensor systems are essential to guided embodiment. In the next sections, we describe sensor systems that can be implemented in ITS for guided embodiment.

Enrichment Effects of Gestures and Pictures on Abstract Words in a Second Language

Laboratory research has demonstrated that multisensory enrichment promotes verbal learning in a foreign language (L2). Enrichment can be done in various ways, e.g., by adding a picture that illustrates the L2 word’s meaning or by the learner performing a gesture to the word (enactment). Most studies have tested enrichment on concrete but not on abstract words. Unlike concrete words, the representation of abstract words is deprived of sensory-motor features. This has been addressed as one of the reasons why abstract words are difficult to remember. Here, we ask whether a brief enrichment training by means of pictures and by self-performed gestures also enhances the memorability of abstract words in L2. Further, we explore which of these two enrichment strategies is more effective. Twenty young adults learned 30 novel abstract words in L2 according to three encoding conditions: (1) reading, (2) reading and pairing the novel word to a picture, and (3) reading and enacting the word by means of a gesture. We measured memory performance in free and cued recall tests, as well as in a visual recognition task. Words encoded with gestures were better remembered in the free recall in the native language (L1). When recognizing the novel words, participants made less errors for words encoded with gestures compared to words encoded with pictures. The reaction times in the recognition task did not differ across conditions. The present findings support, even if only partially, the idea that enactment promotes learning of abstract words and that it is superior to enrichment by means of pictures even after short training.