When to Take a Gesture Seriously: On How We Use and Prioritize Communicative Cues

Individual differences in representational gesture production are associated with cognitive and empathy skills

Substantial individual variation exists in the frequency of gestures produced while speaking. This study investigated the associations between cognitive abilities, empathy levels, and personality traits with the frequency of representational gestures. A cartoon narration task and a social dilemma solving task were used to elicit gestures. Predictor variables were selected based on prior research on individual differences in gesture production and the cognitive and communicative functions of gestures in speech. Our findings revealed that an increased frequency of representational gestures was associated with higher empathy levels in the cartoon narration task. However, in the social dilemma solving task, a higher frequency of representational gestures was associated with lower visuospatial working memory, spatial transformation, and inhibition control abilities. Moreover, no significant relationships were found between verbal working memory, personality traits, and the frequency of representational gestures in either task. These findings suggested that predictor variables for representational gesture production vary depending on the nature of the gesture elicitation task (e.g., spatiomotoric vs. abstract topics). Future research should examine the relationship between individuals' cognitive abilities, empathy and gesture production with across a broader range of topics and in more ecologically valid contexts.

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Addressees Are Sensitive to the Presence of Gesture When Tracking a Single Referent in Discourse

Production studies show that anaphoric reference is bimodal. Speakers can introduce a referent in speech by also using a localizing gesture, assigning a specific locus in space to it. Referring back to that referent, speakers then often accompany a spoken anaphor with a localizing anaphoric gesture (i.e., indicating the same locus). Speakers thus create visual anaphoricity in parallel to the anaphoric process in speech. In the current perception study, we examine whether addressees are sensitive to localizing anaphoric gestures and specifically to the (mis)match between recurrent use of space and spoken anaphora. The results of two reaction time experiments show that, when a single referent is gesturally tracked, addressees are sensitive to the presence of localizing gestures, but not to their spatial congruence. Addressees thus seem to integrate gestural information when processing bimodal anaphora, but their use of locational information in gestures is not obligatory in every discourse context.

The listener automatically uses spatial story representations from the speaker's cohesive gestures when processing subsequent sentences without gestures

This study examined spatial story representations created by speaker's cohesive gestures. Participants were presented with three-sentence discourse with two protagonists. In the first and second sentences, gestures consistently located the two protagonists in the gesture space: one to the right and the other to the left. The third sentence (without gestures) referred to one of the protagonists, and the participants responded with one of the two keys to indicate the relevant protagonist. The response keys were either spatially congruent or incongruent with the gesturally established locations for the two participants. Though the cohesive gestures did not provide any clue for the correct response, they influenced performance: the reaction time in the congruent condition was faster than that in the incongruent condition. Thus, cohesive gestures automatically establish spatial story representations and the spatial story representations remain activated in a subsequent sentence without any gesture.