The role of speech-gesture congruency and delay in remembering action events

Early or synchronized gestures facilitate speech recall-a study based on motion capture data

Introduction

Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.

Methods

To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.

Results

Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.

Discussion

These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

The role of hand gestures in emotion communication: Do type and size of gestures matter?

We communicate emotions in a multimodal way, yet non-verbal emotion communication is a relatively understudied area of research. In three experiments, we investigated the role of gesture characteristics (e.g., type, size in space) on individuals' processing of emotional content. In Experiment 1, participants were asked to rate the emotional intensity of emotional narratives from the videoclips either with iconic or beat gestures. Participants in the iconic gesture condition rated the emotional intensity higher than participants in the beat gesture condition. In Experiment 2, the size of gestures and its interaction with gesture type were investigated in a within-subjects design. Participants again rated the emotional intensity of emotional narratives from the videoclips. Although individuals overall rated narrow gestures more emotionally intense than wider gestures, no effects of gesture type, or gesture size and type interaction were found. Experiment 3 was conducted to check whether findings of Experiment 2 were due to viewing gestures in all videoclips. We compared the gesture and no gesture (i.e., speech only) conditions and showed that there was not a difference between them on emotional ratings. However, we could not replicate the findings related to gesture size of Experiment 2. Overall, these findings indicate the importance of examining gesture's role in emotional contexts and that different gesture characteristics such as size of gestures can be considered in nonverbal communication.

Instructed Hand Movements Affect Students' Learning of an Abstract Concept From Video

Producing content-related gestures has been found to impact students' learning, whether such gestures are spontaneously generated by the learner in the course of problem-solving, or participants are instructed to pose based on experimenter instructions during problem-solving and word learning. Few studies, however, have investigated the effect of (a) performing instructed gestures while learning concepts or (b) producing gestures without there being an implied connection between the gestures and the concepts being learned. The two studies reported here investigate the impact of instructed hand movements on students' subsequent understanding of a concept. Students were asked to watch an instructional video-focused on the concept of statistical model-three times. Two experimental groups were given a secondary task to perform while watching the video, which involved moving their hands to mimic the placement and orientation of red rectangular bars overlaid on the video. Students were told that the focus of the study was multitasking, and that the instructed hand movements were unrelated to the material being learned. In the content-match group the placement of the hands reinforced the concept being explained, and in the content-mismatch group it did not. A control group was not asked to perform a secondary task. In both studies, findings indicate that students in the content-match group performed better on the posttest, and showed less variation in performance, than did students in the content-mismatch group, with control students falling in between. Instructed hand movement-even when presented as an unrelated, secondary task-can affect students' learning of a complex concept.