Toward a more embedded/extended perspective on the cognitive function of gestures

Multimodal language in child-directed versus adult-directed speech

Speakers design their multimodal communication according to the needs and knowledge of their interlocutors, phenomenon known as audience design. We use more sophisticated language (e.g., longer sentences with complex grammatical forms) when communicating with adults compared with children. This study investigates how speech and co-speech gestures change in adult-directed speech (ADS) versus child-directed speech (CDS) for three different tasks. Overall, 66 adult participants (Mage = 21.05, 60 female) completed three different tasks (story-reading, storytelling and address description) and they were instructed to pretend to communicate with a child (CDS) or an adult (ADS). We hypothesised that participants would use more complex language, more beat gestures, and less iconic gestures in the ADS compared with the CDS. Results showed that, for CDS, participants used more iconic gestures in the story-reading task and storytelling task compared with ADS. However, participants used more beat gestures in the storytelling task for ADS than CDS. In addition, language complexity did not differ across conditions. Our findings indicate that how speakers employ different types of gestures (iconic vs beat) according to the addressee's needs and across different tasks. Speakers might prefer to use more iconic gestures with children than adults. Results are discussed according to audience design theory.

Building a mental toolbox: Relationships between strategy choice and sight-singing performance in higher education

Sight-singing is an inescapable component of music training in higher education and is often challenging for students. However, some strategies could help students perform. Yet, the extent to which students can use strategies to improve their sight-singing performance remains unclear. This article asks two questions to fill this gap: (1) Which strategies do students use when sight-singing? (2) Does the application of some types of strategy predict performance? We recruited 56 postsecondary music students and asked them about their musical backgrounds. They then sight-sang a short melody while we recorded their eye movements. After that, we conducted semi-structured retrospective interviews, using eye-movement videos and attention distribution heatmaps to help participants remember the strategies they used. We analyzed the interview transcripts to identify the strategies students used and regrouped them into categories. We extracted seven categories and discovered that using body movements predicted rhythm scores, that using musical knowledge predicted pitch and combined scores, and that relying on automatic skills predicted all dimensions of sight-singing performance. We recommend that aural skills instructors teach strategies explicitly and help students develop robust musical knowledge, as they are required to build strong automatic skills.

Facilitative Effects of Embodied English Instruction in Chinese Children

Research into the lexical quality of word representations suggests that building a strong sound, form, and meaning association is a crucial first step for vocabulary learning. For children who are learning a second language (L2), explicit instruction on word morphology is generally more focused on whole word, rather than sub-lexical, meaning. Though morphological training is emphasized in first language (L1) vocabulary instruction, it is unknown whether this training facilitates L2 word learning through sub-lexical support. To test this, we designed three experimental learning conditions investigating embodied morphological instruction [i.e., hand writing roots (HR), dragging roots (DR), gesturing roots (GR)] to compare against a control condition. One hundred students were randomly assigned to the four experimental groups. Pre- and post-tests examining knowledge of word meanings, forms, and sounds were administered. Results of mixed linear modeling revealed that three embodied morphological instruction on roots enhanced L2 vocabulary learning. Hand writing roots facilitated sound-meaning integration in all category-tasks for accessibility to word form and one task for word sound-form association. By contrast, GR facilitated meaning-based learning integration in two out of three category tasks for word form-meaning association. Chunking and DR facilitated meaning-based integration in one out of three category tasks for word form-meaning association. These results provide evidence that the underlying embodied morphological training mechanism contributes to L2 vocabulary learning during direct instruction. Future directions and implications are discussed.

Socially Extended Scientific Knowledge

A three-tiered account of social cognition is set out-along with the corresponding variety of social knowledge that results from this social cognition-and applied to the special case of scientific collaboration. The first tier is socially-facilitated cognition, which results in socially-facilitated knowledge. This is a form of cognition which, while genuinely social (in that social factors play an important explanatory role in producing the target cognitive success), falls short of socially extended cognition. The second tier is socially extended cognition, which generates socially extended knowledge. This form of cognition is social in the specific sense of the information-processing of other agents forms part of the socially extended cognitive process at issue. It is argued, however, that the core notion of socially extended cognition is individual in nature, in that the target cognitive success is significantly creditable to the socially extended cognitive agency of the individual. Socially extended cognition, in its core sense, thus generates individual knowledge. Finally, there is distributed cognition, which generates distributed knowledge. This is where the cognitive successes produced by a research team are attributable to a group agent rather than to individuals within the team. Accordingly, where this form of social cognition generates knowledge (distributed knowledge), the knowledge is irreducibly group knowledge. It is argued that by making clear this three-tiered structure of social scientific knowledge a prima facie challenge is posed for defenders of distributed scientific cognition and knowledge to explain why this form of social knowledge is being exhibited and not one of the two weaker (and metaphysically less demanding) forms of social knowledge.

Adding and subtracting by hand: Metaphorical representations of arithmetic in spontaneous co-speech gestures

This study investigated the spontaneous co-speech gestures produced by speakers who were talking about the concepts of addition and subtraction in a television news setting. We performed a linguistic and co-speech gesture analysis of expressions related to the concepts of addition (N plus N, addition, add) and subtraction (N minus N, subtraction, subtract). First, we compared the linguistic frequency of these structures across several corpora. Second, we performed a multimodal gesture analysis, drawing data from a television news repository. We analyzed 423 co-speech gestures (169 for subtraction and 254 for addition) in terms of their axis (e.g., lateral, sagittal) and their direction (e.g., leftwards, away from their body). Third, we examined the semantic properties of the direct object that was added or subtracted. There were two main findings. First, low-frequency linguistic expressions were more likely to be accompanied by co-speech gestures. Second, most gestures about addition and subtraction were produced along the lateral or sagittal axes. When people spoke about addition, they tended to produce lateral, rightwards movements or movements away from the body. When people spoke about subtraction, they tended to produce lateral, leftwards movements or movements towards the body. This co-speech gesture data provides evidence that people activate two different metaphors for arithmetic in spontaneous behavior: arithmetic is motion along a path and arithmetic is collecting objects.

Easier Said Than Done? Task Difficulty's Influence on Temporal Alignment, Semantic Similarity, and Complexity Matching Between Gestures and Speech

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches, which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and complexity matching. Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular.

Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources

Speakers use spontaneous hand gestures as they speak and think. These gestures serve many functions for speakers who produce them as well as for listeners who observe them. To date, studies in the gesture literature mostly focused on group-comparisons or the external sources of variation to examine when people use, process, and benefit from using and observing gestures. However, there are also internal sources of variation in gesture use and processing. People differ in how frequently they use gestures, how salient their gestures are, for what purposes they produce gestures, and how much they benefit from using and seeing gestures during comprehension and learning depending on their cognitive dispositions. This review addresses how individual differences in different cognitive skills relate to how people employ gestures in production and comprehension across different ages (from infancy through adulthood to healthy aging) from a functionalist perspective. We conclude that speakers and listeners can use gestures as a compensation tool during communication and thinking that interacts with individuals' cognitive dispositions.

Structural organization of the praxis network predicts gesture production: Evidence from healthy subjects and patients with schizophrenia

Hand gestures are an integral part of social interactions and communication. Several imaging studies in healthy subjects and lesion studies in patients with apraxia suggest the praxis network for gesture production, involving mainly left inferior frontal, posterior parietal and temporal regions. However, little is known about the structural connectivity underlying gesture production. We recruited 41 healthy participants and 39 patients with schizophrenia. All participants performed a gesture production test, the Test of Upper Limb Apraxia, and underwent diffusion tensor imaging. We hypothesized that gesture production is associated with structural network connectivity as well as with tract integrity. We defined the praxis network as an undirected graph comprised of 13 bilateral regions of interest and derived measures of local and global structural connectivity and tract integrity from Finsler geometry. We found an association of gesture deficit with reduced global and local efficiency of the praxis network. Furthermore, reduced tract integrity, for example in the superior longitudinal fascicle, arcuate fascicle or corpus callosum were related to gesture deficits. Our findings contribute to the understanding of structural correlates of gesture production as they first present diffusion tensor imaging data in a combined sample of healthy subjects and a patient cohort with gestural deficits.

Entrainment and Modulation of Gesture-Speech Synchrony Under Delayed Auditory Feedback

Gesture–speech synchrony re‐stabilizes when hand movement or speech is disrupted by a delayed feedback manipulation, suggesting strong bidirectional coupling between gesture and speech. Yet it has also been argued from case studies in perceptual–motor pathology that hand gestures are a special kind of action that does not require closed‐loop re‐afferent feedback to maintain synchrony with speech. In the current pre‐registered within‐subject study, we used motion tracking to conceptually replicate McNeill's (1992) classic study on gesture–speech synchrony under normal and 150 ms delayed auditory feedback of speech conditions (NO DAF vs. DAF). Consistent with, and extending McNeill's original results, we obtain evidence that (a) gesture‐speech synchrony is more stable under DAF versus NO DAF (i.e., increased coupling effect), (b) that gesture and speech variably entrain to the external auditory delay as indicated by a consistent shift in gesture‐speech synchrony offsets (i.e., entrainment effect), and (c) that the coupling effect and the entrainment effect are co‐dependent. We suggest, therefore, that gesture–speech synchrony provides a way for the cognitive system to stabilize rhythmic activity under interfering conditions.

The effects of observing and producing gestures on Japanese word learning

While observing and producing gesture can enhance native language learning, research in foreign word learning is limited. English-speaking university students were verbally presented with Japanese verbs with their English translations in one of three conditions: no gesture, observing an instructor's iconic gestures, or observing and reproducing the instructor's gestures. Participants verbally recalled the words immediately following training and after a one-week delay. Both observing only as well as observing and reproducing gestures at encoding enhanced learning and were equally beneficial for both the short and longer term beyond verbal learning alone. Reproducing the instructor's gestures during learning increased the rate of spontaneous iconic gesture production during recall, though the effects of this spontaneous gesture production at test were inconsistent. Findings emphasise the importance of an instructor's gesture production during foreign language learning, and suggest that motor imagery plays a key role in the beneficial effects of gesture on learning.

Nodding and shaking of the head as simulated approach and avoidance responses

Our recent study within the embodiment perspective showed that the evaluation of true and false information activates the simulation of vertical and horizontal head movements involved in nodding and shaking of the head (Moretti & Greco, 2018). This result was found in an explicit evaluation task where motion detection software was deployed to enable participants to assess a series of objectively true or false statements by moving them with the head vertically and horizontally on a computer screen, under conditions of compatibility and incompatibility between simulated and performed action. This study replicated that experiment, but with subjective statements about liked and disliked food, in both explicit and implicit evaluation tasks. Two experiments, plus one control experiment, were devised to test the presence of a motor-affective compatibility effect (vertical-liked; horizontal-disliked) and whether the motor-semantic compatibility found with objective statements (vertical-true; horizontal-false) could be a sub-effect of a more general and automatic association (vertical-accepted; horizontal-refused). As expected, response times were shorter when statements about liked foods and disliked foods were moved vertically and horizontally respectively by making head movements, even when participants were not explicitly required to evaluate them. In contrast, the truth compatibility effect only occurred in the explicit evaluation task. Overall results support the idea that head-nodding and shaking are simulated approach-avoidance responses. Different aspects of the meaning of these gestures and the practical implications of the study for cognitive and social research are discussed.

The role of gesture as simulated action in reinterpretation of mental imagery

In two experiments, we examined the role of gesture in reinterpreting a mental image. In Experiment 1, we found that participants gestured more about a figure they had learned through manual exploration than about a figure they had learned through vision. This supports claims that gestures emerge from the activation of perception-relevant actions during mental imagery. In Experiment 2, we investigated whether such gestures have a causal role in affecting the quality of mental imagery. Participants were randomly assigned to gesture, not gesture, or engage in a manual interference task as they attempted to reinterpret a figure they had learned through manual exploration. We found that manual interference significantly impaired participants' success on the task. Taken together, these results suggest that gestures reflect mental imaginings of interactions with a mental image and that these imaginings are critically important for mental manipulation and reinterpretation of that image. However, our results suggest that enacting the imagined movements in gesture is not critically important on this particular task.

Are gesture and speech mismatches produced by an integrated gesture-speech system? A more dynamically embodied perspective is needed for understanding gesture-related learning

We observe a tension in the target article as it stresses an integrated gesture-speech system that can nevertheless consist of contradictory representational states, which are reflected by mismatches in gesture and speech or sign. Beyond problems of coherence, this prevents furthering our understanding of gesture-related learning. As a possible antidote, we invite a more dynamically embodied perspective to the stage.

A Feel for Numbers: The Changing Role of Gesture in Manipulating the Mental Representation of an Abacus Among Children at Different Skill Levels

Abacus mental arithmetic involves the skilled acquisition of a set of gestures representing mathematical algorithms to properly manipulate an imaginary abacus. The present study examined how the beneficial effect of abacus co-thought gestures varied at different skill and problem difficulty levels. We compared the mental arithmetic performance of 6- to 8-year-old beginning (N = 57), intermediate (N = 65), and advanced (N = 54) learners under three conditions: a physical abacus, hands-free (spontaneous gesture) mental arithmetic, and hands-restricted mental arithmetic. We adopted a mixed-subject design, with level of difficulty and skill level as the within-subject independent variables and condition as the between-subject independent variable. Our results showed a clear contrast in calculation performance and gesture accuracy among learners at different skill levels. Learners first mastered how to calculate using a physical abacus and later benefitted from using abacus gestures to aid mental arithmetic. Hand movement and gesture accuracy indicated that the beneficial effect of gestures may be related to motor learning. Beginners were proficient with a physical abacus, but performed poorly and had low gesture accuracy during mental arithmetic. Intermediates relied on gestures to do mental arithmetic and had accurate hand movements, but performed more poorly when restricted from gesturing. Advanced learners could perform mental arithmetic with accurate gestures and scored just as well without gesturing. These findings suggest that for intermediate and advanced learners, motor-spatial representation through abacus co-thought gestures may complement visual-spatial representation of a mental abacus to reduce working memory load.

Thinking While Moving or Moving While Thinking - Concepts of Motor-Cognitive Training for Cognitive Performance Enhancement

The demographic change in industrial countries, with increasingly sedentary lifestyles, has a negative impact on mental health. Normal and pathological aging leads to cognitive deficits. This development poses major challenges on national health systems. Therefore, it is necessary to develop efficient cognitive enhancement strategies. The combination of regular physical exercise with cognitive stimulation seems especially suited to increase an individual's cognitive reserve, i.e., his/her resistance to degenerative processes of the brain. Here, we outline insufficiently explored fields in exercise-cognition research and provide a classification approach for different motor-cognitive training regimens. We suggest to classify motor-cognitive training in two categories, (I) sequential motor-cognitive training (the motor and cognitive training are conducted time separated) and (II) simultaneous motor-cognitive training (motor and cognitive training are conducted sequentially). In addition, simultaneous motor-cognitive training may be distinguished based on the specific characteristics of the cognitive task. If successfully solving the cognitive task is not a relevant prerequisite to complete the motor-cognitive task, we would consider this type of training as (IIa) motor-cognitive training with additional cognitive task. In contrast, in ecologically more valid (IIb) motor cognitive training with incorporated cognitive task, the cognitive tasks are a relevant prerequisite to solve the motor-cognitive task. We speculate that incorporating cognitive tasks into motor tasks, rather than separate training of mental and physical functions, is the most promising approach to efficiently enhance cognitive reserve. Further research investigating the influence of motor(-cognitive) exercises with different quantitative and qualitative characteristics on cognitive performance is urgently needed.

Co-thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity

During silent problem solving, hand gestures arise that have no communicative intent. The role of such co-thought gestures in cognition has been understudied in cognitive research as compared to co-speech gestures. We investigated whether gesticulation during silent problem solving supported subsequent performance in a Tower of Hanoi problem-solving task, in relation to visual working-memory capacity and task complexity. Seventy-six participants were assigned to either an instructed gesture condition or a condition that allowed them to gesture, but without explicit instructions to do so. This resulted in three gesture groups: (1) non-gesturing; (2) spontaneous gesturing; (3) instructed gesturing. In line with the embedded/extended cognition perspective on gesture, gesturing benefited complex problem-solving performance for participants with a lower visual working-memory capacity, but not for participants with a lower spatial working-memory capacity.

Embodied learning: introducing a taxonomy based on bodily engagement and task integration

Research on learning and education is increasingly influenced by theories of embodied cognition. Several embodiment-based interventions have been empirically investigated, including gesturing, interactive digital media, and bodily activity in general. This review aims to present the most important theoretical foundations of embodied cognition and their application to educational research. Furthermore, we critically review recent research concerning the effectiveness of embodiment interventions and develop a taxonomy to more properly characterize research on embodied cognition. The main dimensions of this taxonomy are bodily engagement (i.e. how much bodily activity is involved) and task integration (i.e. whether bodily activities are related to a learning task in a meaningful way or not). By locating studies on the 2 × 2 grid resulting from this taxonomy and assessing the corresponding learning outcomes, we identify opportunities, problems, and challenges of research on embodied learning.

Links between Gestures and Multisensory Processing: Individual Differences Suggest a Compensation Mechanism

Speech-associated gestures represent an important communication modality. However, individual differences in the production and perception of gestures are not well understood so far. We hypothesized that the perception of multisensory action consequences might play a crucial role. Verbal communication involves continuous calibration of audio–visual information produced by the speakers. The effective production and perception of gestures supporting this process could depend on the given capacities to perceive multisensory information accurately. We explored the association between the production and perception of gestures and the monitoring of multisensory action consequences in a sample of 31 participants. We applied a recently introduced gesture scale to assess self-reported gesture production and perception in everyday life situations. In the perceptual experiment, we presented unimodal (visual) and bimodal (visual and auditory) sensory outcomes with various delays after a self-initiated (active) or externally generated (passive) button press. Participants had to report whether they detected a delay between the button press and the visual stimulus. We derived psychometric functions for each condition and determined points of subjective equality, reflecting detection thresholds for delays. Results support a robust link between gesture scores and detection thresholds. Individuals with higher detection thresholds (lower performance) reported more frequent gesture production and perception and furthermore profited more from multisensory information in the experimental task. We propose that our findings indicate a compensational function of multisensory processing as a basis for individual differences in both action outcome monitoring and gesture production and perception in everyday life situations.

When to Take a Gesture Seriously: On How We Use and Prioritize Communicative Cues

When people talk, their speech is often accompanied by gestures. Although it is known that co-speech gestures can influence face-to-face communication, it is currently unclear to what extent they are actively used and under which premises they are prioritized to facilitate communication. We investigated these open questions in two experiments that varied how pointing gestures disambiguate the utterances of an interlocutor. Participants, whose event-related brain responses were measured, watched a video, where an actress was interviewed about, for instance, classical literature (e.g., Goethe and Shakespeare). While responding, the actress pointed systematically to the left side to refer to, for example, Goethe, or to the right to refer to Shakespeare. Her final statement was ambiguous and combined with a pointing gesture. The P600 pattern found in Experiment 1 revealed that, when pointing was unreliable, gestures were only monitored for their cue validity and not used for reference tracking related to the ambiguity. However, when pointing was a valid cue (Experiment 2), it was used for reference tracking, as indicated by a reduced N400 for pointing. In summary, these findings suggest that a general prioritization mechanism is in use that constantly monitors and evaluates the use of communicative cues against communicative priors on the basis of accumulated error information.

Action Contribution to Competence Judgments: The Use of the Journey Schema

The current research considered the question of how performing an action, or merely preparing the body for action, can have an impact on social judgments related to person perception. Participants were asked to ascribe competence and warmth characteristics to a target person by reading a metaphoric text while their body was manipulated to be prepared for the processing of action-congruent information. In Experiment 1, participants whose forward body action matched the metaphoric action described in the text ascribed more competence characteristics to a politician than did control participants. In Experiment 2, participants whose body was merely prepared for forward movement also ascribed more competence characteristics to a politician than did control participants. In addition, the data from Experiment 2 ruled out an alternative non-embodied explanation (i.e., that effect is due to basic associative processes) grounded in the existing literatures on attitudes by demonstrating that body manipulation had no effect on competence when a non-metaphoric text was used. Finally, no evidence was found that body manipulation affects warmth judgments. These studies converge in demonstrating that forward body movements enhance the favorability of competence judgments when these match the metaphoric forward movements described by text.

Mechanisms of embodiment

This paper is a critical review of recent studies demonstrating the mechanism of sensorimotor simulation in different cognitive domains. Empirical studies that specify conditions under which embodiment occurs in different domains will be discussed and evaluated. Examples of relevant domains are language comprehension (Tucker and Ellis, 1998), autobiographical memory (Dijkstra et al., 2007), gestures (Alibali et al., 2014), facial mimicry (Stel and Vonk, 2010), and problem solving (Wiemers et al., 2014). The focus of the review is on supporting claims regarding sensorimotor simulation as well as on factors that modulate dynamic relationships between sensorimotor components in action and cognitive domains, such as expertise (Boschker et al., 2002). This discussion takes place within the context of currently debated issues, specifically the need to specify the underlying mechanisms of embodied representations (Zwaan, 2014; Körner et al., 2015).

Phonological similarity affects production of gestures, even in the absence of overt speech

Highlights

Does phonological similarity affect gesture production in the absence of speech?

Participants produced gestures from pictures with no words presented or spoken.

Same pictures and gestures but different training labels were used.

Phonologically similar labels led to more errors in subsequent gestures.

Thus, phonological similarity affects gesture production in the absence of speech.

Are manual gestures affected by inner speech? This study tested the hypothesis that phonological form influences gesture by investigating whether phonological similarity between words that describe motion gestures creates interference for production of those gestures in the absence of overt speech. Participants learned to respond to a picture of a bottle by gesturing to open the bottle's cap, and to a picture of long hair by gesturing to twirl the hair. In one condition, the gestures were introduced with phonologically-similar labels “twist” and “twirl” (similar condition), while in the other condition, they were introduced with phonologically-dissimilar labels “unscrew” and “twirl” (dissimilar condition). During the actual experiment, labels were not produced and participants only gestured by looking at pictures. In both conditions, participants also gestured to a control pair that was used as a baseline. Participants made significantly more errors on gestures in the similar than dissimilar condition after correction for baseline differences. This finding shows the influence of phonology on gesture production in the absence of overt speech and poses new constraints on the locus of the interaction between language and gesture systems.

Rethinking situated and embodied social psychology

This article aims to explore the scope of a Situated and Embodied Social Psychology (ESP). At first sight, social cognition seems embodied cognition par excellence. Social cognition is first and foremost a supra-individual, interactive, and dynamic process (Semin & Smith, 2013). Radical approaches in Situated/Embodied Cognitive Science (Enactivism) claim that social cognition consists in an emergent pattern of interaction between a continuously coupled organism and the (social) environment; it rejects representationalist accounts of cognition (Hutto & Myin, 2013). However, mainstream ESP (Barsalou, 1999, 2008) still takes a rather representation-friendly approach that construes embodiment in terms of specific bodily formatted representations used (activated) in social cognition. We argue that mainstream ESP suffers from vestiges of theoretical solipsism, which may be resolved by going beyond internalistic spirit that haunts mainstream ESP today.

A speaker's gesture style can affect language comprehension: ERP evidence from gesture-speech integration

In face-to-face communication, speech is typically enriched by gestures. Clearly, not all people gesture in the same way, and the present study explores whether such individual differences in gesture style are taken into account during the perception of gestures that accompany speech. Participants were presented with one speaker that gestured in a straightforward way and another that also produced self-touch movements. Adding trials with such grooming movements makes the gesture information a much weaker cue compared with the gestures of the non-grooming speaker. The Electroencephalogram was recorded as participants watched videos of the individual speakers. Event-related potentials elicited by the speech signal revealed that adding grooming movements attenuated the impact of gesture for this particular speaker. Thus, these data suggest that there is sensitivity to the personal communication style of a speaker and that affects the extent to which gesture and speech are integrated during language comprehension.

Inconsistent use of gesture space during abstract pointing impairs language comprehension

Pointing toward concrete objects is a well-known and efficient communicative strategy. Much less is known about the communicative effectiveness of abstract pointing where the pointing gestures are directed to “empty space.” McNeill's (2003) observations suggest that abstract pointing can be used to establish referents in gesture space, without the referents being physically present. Recently, however, it has been shown that abstract pointing typically provides redundant information to the uttered speech thereby suggesting a very limited communicative value (So et al., 2009). In a first approach to tackle this issue we were interested to know whether perceivers are sensitive at all to this gesture cue or whether it is completely discarded as irrelevant add-on information. Sensitivity to for instance a gesture-speech mismatch would suggest a potential communicative function of abstract pointing. Therefore, we devised a mismatch paradigm in which participants watched a video where a female was interviewed on various topics. During her responses, she established two concepts in space using abstract pointing (e.g., pointing to the left when saying Donald, and pointing to the right when saying Mickey). In the last response to each topic, the pointing gesture accompanying a target word (e.g., Donald) was either consistent or inconsistent with the previously established location. Event related brain potentials showed an increased N400 and P600 when gesture and speech referred to different referents, indicating that inconsistent use of gesture space impairs language comprehension. Abstract pointing was found to influence comprehension even though gesture was not crucial to understanding the sentences or conducting the experimental task. These data suggest that a referent was retrieved via abstract pointing and that abstract pointing can potentially be used for referent indication in a discourse. We conclude that abstract pointing has a potential communicative function.