Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources

Learners' Spontaneous Gesture Before a Math Lesson Predicts the Efficacy of Seeing Versus Doing Gesture During the Lesson

Gestures-hand movements that accompany speech and express ideas-can help children learn how to solve problems, flexibly generalize learning to novel problem-solving contexts, and retain what they have learned. But does it matter who is doing the gesturing? We know that producing gesture leads to better comprehension of a message than watching someone else produce gesture. But we do not know how producing versus observing gesture impacts deeper learning outcomes such as generalization and retention across time. Moreover, not all children benefit equally from gesture instruction, suggesting that there are individual differences that may play a role in who learns from gesture. Here, we consider two factors that might impact whether gesture leads to learning, generalization, and retention after mathematical instruction: (1) whether children see gesture or do gesture and (2) whether a child spontaneously gestures before instruction when explaining their problem-solving reasoning. For children who spontaneously gestured before instruction, both doing and seeing gesture led to better generalization and retention of the knowledge gained than a comparison manipulative action. For children who did not spontaneously gesture before instruction, doing gesture was less effective than the comparison action for learning, generalization, and retention. Importantly, this learning deficit was specific to gesture, as these children did benefit from doing the comparison manipulative action. Our findings are the first evidence that a child's use of a particular representational format for communication (gesture) directly predicts that child's propensity to learn from using the same representational format.

Through Thick and Thin: Gesture and Speech Remain as an Integrated System in Atypical Development

Gesture and speech are tightly linked and form a single system in typical development. In this review, we ask whether and how the role of gesture and relations between speech and gesture vary in atypical development by focusing on two groups of children: those with peri- or prenatal unilateral brain injury (children with BI) and preterm born (PT) children. We describe the gestures of children with BI and PT children and the relations between gesture and speech, as well as highlight various cognitive and motor antecedents of the speech-gesture link observed in these populations. We then examine possible factors contributing to the variability in gesture production of these atypically developing children. Last, we discuss the potential role of seeing others' gestures, particularly those of parents, in mediating the predictive relationships between early gestures and upcoming changes in speech. We end the review by charting new areas for future research that will help us better understand the robust roles of gestures for typical and atypically-developing child populations.

Give your ideas a hand: the role of iconic hand gestures in enhancing divergent creative thinking

Hand gestures play an integral role in multimodal language and communication. Even though the self-oriented functions of gestures, such as activating a speaker's lexicon and maintaining visuospatial imagery, have been emphasized, gestures' functions in creative thinking are not well-established. In the current study, we investigated the role of iconic gestures in verbal divergent thinking-a creative thinking process related to generating many novel ideas. Based on previous findings, we hypothesized that iconic gesture use would facilitate divergent thinking in young adults, especially those with high mental imagery skills. Participants performed Guildford's Alternative Uses Task in a gesture-spontaneous and in a gesture-encouraged condition. We measured fluency (number of ideas), originality (uniqueness of ideas), flexibility (number of idea categories), and elaboration (number of details) in divergent thinking. The results showed that producing iconic gestures in the gesture-encouraged condition positively predicted fluency, originality, and elaboration. In the gesture-spontaneous condition, producing iconic gestures also positively predicted elaboration but negatively predicted flexibility. Mental imagery skills did not interact with the effects of gestures on divergent thinking. These results suggest that iconic gestures are a promising candidate for enhancing almost all aspects of divergent thinking. Overall, the current study adds a new dimension to the self-oriented function of iconic gestures, that is, their contribution to creative thinking.

Individual differences in representational gesture production are associated with cognitive and empathy skills

Substantial individual variation exists in the frequency of gestures produced while speaking. This study investigated the associations between cognitive abilities, empathy levels, and personality traits with the frequency of representational gestures. A cartoon narration task and a social dilemma solving task were used to elicit gestures. Predictor variables were selected based on prior research on individual differences in gesture production and the cognitive and communicative functions of gestures in speech. Our findings revealed that an increased frequency of representational gestures was associated with higher empathy levels in the cartoon narration task. However, in the social dilemma solving task, a higher frequency of representational gestures was associated with lower visuospatial working memory, spatial transformation, and inhibition control abilities. Moreover, no significant relationships were found between verbal working memory, personality traits, and the frequency of representational gestures in either task. These findings suggested that predictor variables for representational gesture production vary depending on the nature of the gesture elicitation task (e.g., spatiomotoric vs. abstract topics). Future research should examine the relationship between individuals' cognitive abilities, empathy and gesture production with across a broader range of topics and in more ecologically valid contexts.

Multimodal language in child-directed versus adult-directed speech

Speakers design their multimodal communication according to the needs and knowledge of their interlocutors, phenomenon known as audience design. We use more sophisticated language (e.g., longer sentences with complex grammatical forms) when communicating with adults compared with children. This study investigates how speech and co-speech gestures change in adult-directed speech (ADS) versus child-directed speech (CDS) for three different tasks. Overall, 66 adult participants (Mage = 21.05, 60 female) completed three different tasks (story-reading, storytelling and address description) and they were instructed to pretend to communicate with a child (CDS) or an adult (ADS). We hypothesised that participants would use more complex language, more beat gestures, and less iconic gestures in the ADS compared with the CDS. Results showed that, for CDS, participants used more iconic gestures in the story-reading task and storytelling task compared with ADS. However, participants used more beat gestures in the storytelling task for ADS than CDS. In addition, language complexity did not differ across conditions. Our findings indicate that how speakers employ different types of gestures (iconic vs beat) according to the addressee's needs and across different tasks. Speakers might prefer to use more iconic gestures with children than adults. Results are discussed according to audience design theory.

Individual differences in working memory predict the efficacy of experimenter-manipulated gestures in first-grade children

Why is instructional gesture ineffective in some contexts? And what is it about learners that predicts whether they will learn from gestures? This between-subjects linear measurement training study compares gesture instruction to two controls-operant action and transient action-in a diverse sample of first-grade students (N = 174, Mage  = 7.01 years; Nfemale  = 84; Nmale  = 90, 10% Latinx-identified; 70% White; 6% Black; 6% Asian; 18% multiple racial categories, Mincome  = $59,750, SDincome  ≈ $25,000; data collected 03/16-03/19). Results show that instructor-manipulated gestures may be less effective than demonstrative actions in part because they are iterative and do not leave a lasting trace. Verbal working memory, but not spatial, positively predicted an ability to learn from gesture and transient action in children with the lowest context knowledge.

Gestures speed up responses to questions

Most language use occurs in face-to-face conversation, which involves rapid turn-taking. Seeing communicative bodily signals in addition to hearing speech may facilitate such fast responding. We tested whether this holds for co-speech hand gestures by investigating whether these gestures speed up button press responses to questions. Sixty native speakers of Dutch viewed videos in which an actress asked yes/no-questions, either with or without a corresponding iconic hand gesture. Participants answered the questions as quickly and accurately as possible via button press. Gestures did not impact response accuracy, but crucially, gestures sped up responses, suggesting that response planning may be finished earlier when gestures are seen. How much gestures sped up responses was not related to their timing in the question or their timing with respect to the corresponding information in speech. Overall, these results are in line with the idea that multimodality may facilitate fast responding during face-to-face conversation.

Preschool children's discourse competence in different genres and how it relates to iconic gestures

Based on the linguistic analysis of game explanations and retellings, the paper's goal is to investigate the relation of preschool children's situated discourse competence and iconic gestures in different communicative genres, focussing on reinforcing and supplementary speech-gesture-combinations. To this end, a method was developed to evaluate discourse competence as a context-sensitive and interactively embedded phenomenon. The so-called GLOBE-model was adapted to assess discourse competence in relation to interactive scaffolding. The findings show clear links between the children's competence and their parents' scaffolding. We suggest this to be evidence of a fine-tuned interactive support system. The results also indicate strong relations between higher discourse competence and increased frequency of iconic gestures. This applies in particular to reinforcing gestures. The results are interpreted as a confirmation that the speech-gesture system undergoes systematic changes during early childhood, and that gesturing becomes more iconic - and thus more communicative - when discourse competence is growing.

Hands of confidence: When gestures increase confidence in spatial problem-solving

This study examined whether the metacognitive system monitors the potential positive effects of gestures on spatial thinking. Participants (N = 59, 31F, Mage = 21.67) performed a mental rotation task, consisting of 24 problems varying in difficulty, and they evaluated their confidence in their answers to problems in either gesture or control conditions. The results revealed that performance and confidence were higher in the gesture condition, in which the participants were asked to use their gestures during problem-solving, compared with the control condition, extending the literature by evidencing gestures' role in metacognition. Yet, the effect was only evident for females, who already performed worse than males, and when the problems were difficult. Encouraging gestures adversely affected performance and confidence in males. Such results suggest that gestures selectively influence cognition and metacognition and highlight the importance of task-related (i.e., difficulty) and individual-related variables (i.e., sex) in elucidating the links between gestures, confidence, and spatial thinking.

The role of hand gestures in emotion communication: Do type and size of gestures matter?

We communicate emotions in a multimodal way, yet non-verbal emotion communication is a relatively understudied area of research. In three experiments, we investigated the role of gesture characteristics (e.g., type, size in space) on individuals' processing of emotional content. In Experiment 1, participants were asked to rate the emotional intensity of emotional narratives from the videoclips either with iconic or beat gestures. Participants in the iconic gesture condition rated the emotional intensity higher than participants in the beat gesture condition. In Experiment 2, the size of gestures and its interaction with gesture type were investigated in a within-subjects design. Participants again rated the emotional intensity of emotional narratives from the videoclips. Although individuals overall rated narrow gestures more emotionally intense than wider gestures, no effects of gesture type, or gesture size and type interaction were found. Experiment 3 was conducted to check whether findings of Experiment 2 were due to viewing gestures in all videoclips. We compared the gesture and no gesture (i.e., speech only) conditions and showed that there was not a difference between them on emotional ratings. However, we could not replicate the findings related to gesture size of Experiment 2. Overall, these findings indicate the importance of examining gesture's role in emotional contexts and that different gesture characteristics such as size of gestures can be considered in nonverbal communication.

Examining Phenotypical Heterogeneity and its Underlying Factors in Gesture Skills of Chinese Autistic Children: Clustering Analysis

PURPOSE

The heterogeneity of autism is well documented, but few studies have studied the heterogeneity of gesture production ability in autistic children. The present study aimed to identify subgroups of autistic children who displayed heterogeneous gesture production abilities and explore the underlying factors, including autism characteristics, intellectual ability, and language ability, that were associated with the heterogeneity.

METHODS

A total of 65 Chinese autistic children (mean age = 5;3) participated. Their autism characteristics and intellectual ability were assessed by standardized measurements. Language output and gesture production were captured from a parent-child interaction task.

RESULTS

We conducted a hierarchical cluster analysis and identified four distinct clusters. Cluster 1 and Cluster 2 both had low gesture production whereas Cluster 3 and Cluster 4 had high gesture production. Both Clusters 1 and 2 had relatively strong autism characteristics, in comparison to Clusters 3 and 4.

CONCLUSIONS

Our findings revealed that children with stronger autism characteristics may gesture less often than those with weaker characteristics. However, the relationship between language ability and intellectual ability and gesture production was not clear. These findings shed light on the directions of intervention on gesture production for autistic children, especially those with stronger autism characteristics.

Narrative Abilities at Age 3 Are Associated Positively With Gesture Accuracy but Negatively With Gesture Rate

PURPOSE

Though the frequency of gesture use by infants has been related to the development of different language abilities in the initial stages of language acquisition, less is known about whether this frequency (or "gesture rate") continues to correlate with language measures in later stages of language acquisition, or whether the relation to language skills also depends on the accuracy with which such gestures are produced (or reproduced). This study sets out to explore whether preschoolers' narrative abilities are related to these two variables, namely, gesture rate and gesture accuracy.

METHOD

A total of 31 typically developing 3- to 4-year-old children participated in a multimodal imitation task, a context-based gesture elicitation task, and a narrative retelling task.

RESULTS

Results showed that there was a significant positive correlation between the children's narrative scores and their gesture accuracy scores, whereas higher rates of gesture use did not correlate with higher levels of narrative skill. Further multimodal regression analysis confirmed that gesture accuracy was a positive predictor of narrative performance, and moreover, showed that gesture rate was a negative predictor.

CONCLUSIONS

The fact that both gesture accuracy and gesture rate are strongly and differently linked to oral language abilities supports the claim that language and gesture are highly complex systems, and that complementary measures of gesture performance can help us assess with greater granularity the relationship between gesture and language development. These findings highlight the need to use gesture during clinical assessments as an informative indicator of language development and suggest that future research should further investigate the value of multimodal programs in the treatment of language and communication disorders.

The role of gestures in autobiographical memory

Speakers employ co-speech gestures when thinking and speaking; however, gesture's role in autobiographical episodic representations is not known. Based on the gesture-for-conceptualization framework, we propose that gestures, particularly representational ones, support episodic event representations by activating existing episodic elements and causing new ones to be formed in the autobiographical recollections. These gestures may also undertake information-chunking roles to allow for further processing during remembering, such as a sense of recollective experience. Participants (N = 41) verbally narrated three events (a past autobiographical, a future autobiographical, and a non-autobiographical event) and then rated their phenomenological characteristics. We found that, even though gesture use was not different across the three event conditions, representational gestures were positively associated with the episodic event details as well as their recollective quality within the past autobiographical event narratives. These associations were not observed in future event narrations. These findings suggest that gestures are potentially instrumental in the retrieval of details in autobiographical memories.

Comparing mnemonic effects of iconic gestures and pictures on word memory

Previous studies using intermodal semantic priming have found that gestures improve language memory. In the present study, we ask whether the inherent characteristics of representative gestures (iconic gestures) facilitate word memory, or it is simply the semantic content shared with the words. Two analogous experiments were carried out presenting iconic gestures, pictures, or null primes to target words (nouns and action verbs). In Experiment 1, participants performed a free word recall task. In Experiment 2, the task was one of recognition. The results showed that participants recalled (Experiment 1) an equivalent number of words preceded by gestures or pictures compared with words alone, with no prime. However, a significantly higher number of words were recognised (Experiment 2) when they were primed by iconic gestures compared with the other two conditions, an advantage also found in reaction times (RTs) and both effects being larger in verbs than in nouns. These findings are discussed regarding the differences between recall and recognition processes as well as the particular characteristics of representative gestures.

A Hands-On Approach to Learning: Gesture Production During Encoding and its Effect on Narrative Recall

Research has shown that gesture production supports learning across a number of tasks. It is unclear, however, whether gesture production during encoding can support narrative recall, who gesture production benefits most, and whether certain types of gestures are more beneficial than others. This study, therefore, investigated the effect of gesture production during the encoding of a narrative on subsequent narrative recall, and whether individuals' levels of verbal and nonverbal memory moderated this effect. Additionally, this study investigated whether producing certain types of gestures during encoding was more beneficial than others. Participants (N = 90, M_age = 20.43) read aloud a narrative while under instruction to produce gestures, under no specific instruction to produce gestures, or were required to keep their hands behind their back to prevent them from gesturing. Participants completed measures assessing verbal and nonverbal memory. While gesture production during encoding benefitted narrative recall (as measured through specific questions), verbal memory moderated the effect, such that gesture production was more beneficial for individuals with higher than lower verbal memory. Furthermore, producing representational gestures during encoding benefitted recall of points in the narrative at which those gestures were produced, while beat gestures had no effect. Findings have implications for understanding the mechanisms underlying the links between gesture and learning, as well as practical implications in instructional settings.

A cross-sectional study to assess pragmatic strengths and weaknesses in healthy ageing

Background

Ageing refers to the natural and physiological changes that individuals experience over the years. This process also involves modifications in terms of communicative-pragmatics, namely the ability to convey meanings in social contexts and to interact with other people using various expressive means, such as linguistic, extralinguistic and paralinguistic aspects of communication. Very few studies have provided a complete assessment of communicative-pragmatic performance in healthy ageing.

Methods

The aim of this study was to comprehensively assess communicative-pragmatic ability in three samples of 20 (N = 60) healthy adults, each belonging to a different age range (20–40, 65–75, 76–86 years old) and to compare their performance in order to observe any potential changes in their ability to communicate. We also explored the potential role of education and sex on the communicative-pragmatic abilities observed. The three age groups were evaluated with a between-study design by means of the Assessment Battery for Communication (ABaCo), a validated assessment tool characterised by five scales: linguistic, extralinguistic, paralinguistic, contextual and conversational.

Results

The results indicated that the pragmatic ability assessed by the ABaCo is poorer in older participants when compared to the younger ones (main effect of age group: F(2,56) = 9.097; p < .001). Specifically, significant differences were detected in tasks on the extralinguistic, paralinguistic and contextual scales. Whereas the data highlighted a significant role of education (F(1,56) = 4.713; p = .034), no sex-related differences were detected.

Conclusions

Our results suggest that the ageing process may also affect communicative-pragmatic ability and a comprehensive assessment of the components of such ability may help to better identify difficulties often experienced by older individuals in their daily life activities.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-022-03304-z.

Gesture Development in Chinese-Speaking Preschool Children With Autism and the Roles of Parental Input and Child-Based Factors

PURPOSE

Children with autism are found to have delayed and heterogeneous gesture abilities. It is important to understand the growth of gesture abilities and the underlying factors affecting its growth. Addressing these issues can help to design effective intervention programs.

METHOD

Thirty-five Chinese-speaking preschoolers with autism spectrum disorder (M _age = 4.89 years, SD = 0.91; four girls) participated in four play sessions with their parents over 9 months. Their child-based factors including autism severity, intellectual functioning, and expressive language abilities were assessed. The gestures (deictic, iconic, and conventional) of the children and their parents were coded. Growth curve analyses were conducted to examine individual growth trajectories and the roles of child-based factors and parental input in shaping the children's gesture development.

RESULTS

Child-based factors and parental input predicted gesture development differently. Parents' gestures positively predicted their children's gestures of the same type. Autism severity negatively predicted iconic and conventional gestures. Overall growth was found in deictic rather than iconic and conventional gestures. Subgroup variation was also found. Specifically, children with better expressive language ability showed a decrease in deictic gestures. An increase in iconic and conventional gestures was found in children with more severe autism and those with poorer expressive language ability and intellectual functioning, respectively.

CONCLUSIONS

Different types of gestures may have different growth trajectories and be predicted by different child-based factors. Particular attention should be given to children who never produced iconic gestures, which is more challenging and may not develop over a short period, and hence require direct instruction.

Aging, Gesture Production, and Disfluency in Speech: A Comparison of Younger and Older Adults

Age-related changes are observed in the speech and gestures of neurotypical individuals. Older adults are more disfluent in speech and use fewer representational gestures (e.g., holding two hands close to each other to mean small), compared to younger adults. Using gestures, especially representational gestures, is common in difficult tasks to aid the conceptualization process and to facilitate lexical access. This study investigates how aging can affect gesture production and the co-occurrence between gesture and speech disfluency. We elicited speech and gesture samples from younger and older adults (N = 60) by using a painting description task that provided concrete and abstract contexts. Results indicated that albeit the two age groups revealed comparable overall speech disfluency and gesture rates, they differed in terms of how their disfluencies and gestures were distributed across specific categories. Moreover, the proportion of speech disfluencies that occur with a gesture was significantly higher for younger than older adults. However, the two age groups were comparable in terms of the proportion of gestures that were accompanied by a speech disfluency. These findings suggest that younger adults' language production system might be better at benefiting from other modalities, that is, gesture, to resolve temporary problems in speech planning. However, from a gesture perspective, it might be difficult to differentiate between gestures' self-oriented and communicative functions and understand their role in speech facilitation. Focusing on specific cases where speech disfluency and gestures co-occur and considering individual differences might bring insight into multimodal communication.

The role of caregiver gestures and gesture-related responses of toddlers with autism spectrum disorder

Autism spectrum disorder (ASD) is characterized by social communicative abnormalities. Deficits and delays in gestural communication are among the early deficits of ASD and also a major social modality in early caregiver-toddler interaction. Caregiver gestures have an important role in the cognitive and social development of children with ASD. Thus, it is urgent to further explore the role of caregiver gestures in early caregiver-toddler interaction. In this cross-sectional study, we observed the caregivers' gestures and responses of toddlers aged between 18 and 24 months during play (ASD = 44, TD = 29) and dining activities (ASD = 34, TD = 27). By observing the different frequencies and patterns of gestures by the caregiver-child interaction and the different proportions of children's responses to the caregiver's gestures, we found that, compared to caregivers of typically developing toddlers, caregivers of toddlers with ASD had fewer synchronized gestures and more unsynchronized gestures in the play activity and more supplementary gestures in dining activity. Toddlers with ASD produced more social responses to caregivers' synchronized gestures, whereas the use of synchronized gestures by the caregivers in caregiver-toddler interaction had a positive influence on social responses to toddlers with ASD. The findings suggest that effective use of gestures by caregivers during caregiver-toddler activities can improve children's social responses.

Production and comprehension of co-speech gestures in Parkinson's disease

This study examined how impairments in sensorimotor abilities of individuals with Parkinson's Disease (PD) can be related to the use and understanding of co-speech hand gestures involving literal and figurative actions. We tested individuals with PD (n = 18, 12 males, M_age = 56.5, SD_age = 8.16, PD duration since onset: M = 5.36 years, SD = 3.51, Hoehn and Yahr Scale:M_H&Y = 2.09, SD_H&Y = 0.50) and age- and education-matched neurotypical controls (n = 18, 14 males, M_age = 56.61, SD_age = 8.88) with two experimental tasks. In the gesture production task, participants retold the narratives presented to them in a written format. In the gesture comprehension task, participants were asked to match a gesture with a novel verb in literal and figurative sentence contexts. Results showed that patients with PD gestured significantly less than the neurotypical controls. No group differences were found for the type of gesture use. Individuals with PD performed worse than controls on matching gestures with novel verbs, particularly for figurative meanings. Individuals' severity in the disease negatively correlated with their performance for these figurative novel verb-gesture matches. The performances in the two tasks did not correlate. These findings suggest that problems in sensorimotor abilities resulting from PD can influence overall gesture production and gesture comprehension, providing further evidence on the relations between PD and the impaired use of multimodal language.

The Relation Between Cognitive Abilities and the Distribution of Semantic Features Across Speech and Gesture in 4-year-olds

When young children learn to use language, they start to use their hands in co-verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) analysis to characterize combinations of speech and gesture meaning and related them to the child's visual-spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON-R 2 1 2 - 7 test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities.

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.