Functional neuroanatomy of gesture-speech integration in children varies with individual differences in gesture processing

Exploring Individual Differences: A Case for Measuring Children's Spontaneous Gesture Production as a Predictor of Learning From Gesture Instruction

Decades of research have established that learners benefit when instruction includes hand gestures. This benefit is seen when learners watch an instructor gesture, as well as when they are taught or encouraged to gesture themselves. However, there is substantial individual variability with respect to this phenomenon-not all individuals benefit equally from gesture instruction. In the current paper, we explore the sources of this variability. First, we review the existing research on individual differences that do or do not predict learning from gesture instruction, including differences that are either context-dependent (linked to the particular task at hand) or context-independent (linked to the learner across multiple tasks). Next, we focus on one understudied measure of individual difference: the learner's own spontaneous gesture rate. We present data showing rates of "non-gesturers" across a number of studies and we provide theoretical motivation for why this is a fruitful area for future research. We end by suggesting ways in which research on individual differences will help gesture researchers to further refine existing theories and develop specific predictions about targeted gesture intervention for all kinds of learners.

The role of hand gestures in emotion communication: Do type and size of gestures matter?

We communicate emotions in a multimodal way, yet non-verbal emotion communication is a relatively understudied area of research. In three experiments, we investigated the role of gesture characteristics (e.g., type, size in space) on individuals' processing of emotional content. In Experiment 1, participants were asked to rate the emotional intensity of emotional narratives from the videoclips either with iconic or beat gestures. Participants in the iconic gesture condition rated the emotional intensity higher than participants in the beat gesture condition. In Experiment 2, the size of gestures and its interaction with gesture type were investigated in a within-subjects design. Participants again rated the emotional intensity of emotional narratives from the videoclips. Although individuals overall rated narrow gestures more emotionally intense than wider gestures, no effects of gesture type, or gesture size and type interaction were found. Experiment 3 was conducted to check whether findings of Experiment 2 were due to viewing gestures in all videoclips. We compared the gesture and no gesture (i.e., speech only) conditions and showed that there was not a difference between them on emotional ratings. However, we could not replicate the findings related to gesture size of Experiment 2. Overall, these findings indicate the importance of examining gesture's role in emotional contexts and that different gesture characteristics such as size of gestures can be considered in nonverbal communication.

Coordinated social communication in toddlers with and without autism spectrum disorder during a home observation

BACKGROUND & AIMS

Social communication and language skills have been found to be important predictors of long-term outcomes in individuals with autism spectrum disorder (ASD). However, the development of coordinated social communication (i.e., gestures and sounds or words) remains relatively understudied in young children with ASD and developmental delays (DD). This study used a prospective, longitudinal design and granular observational coding to document the coordination of gestures, sounds, and words in a large, heterogeneous sample of toddlers identified with ASD, DD, or typical development (TD) during a naturalistic home observation. Specific aims were: (1) to compare rates per minute and proportions of coordinated child communicative acts across groups; (2) to examine concurrent relationships between coordinated communication and measures of social communication and autism symptoms; and (3) to examine prospective relationships between coordinated communication, receptive and expressive language skills, and autism symptoms collected at 3 years of age.

METHODS

At a mean age of 20.3 months (SD = 2.0), 211 children (n_ASD = 121; n_DD = 46; n_TD = 44) participated in everyday activities with a parent during an hourlong home observation. Rates per minute and proportions of gestures, sounds and words, as well as temporally overlapping gesture + sound, gesture + word, and gesture + phrase combinations, were compared using one-way ANOVA. Pearson product moment correlations between coordinated communicative acts and measures of social communication, language, and autism symptoms were examined.

RESULTS

On average, children with ASD used sounds and gesture + sounds at significantly lower rates than DD and TD groups, who did not differ. Children with ASD and DD coordinated gesture + single words and gesture + phrases at significantly lower rates than the TD group. Groups did not differ with respect to the rate per minute at which they used gestures alone. Children with TD used a smaller proportion of sounds alone and higher proportions of words and phrases, with and without coordinated gestures, than ASD and DD groups. Children with ASD and DD used a significantly higher proportion of gestures alone than children with TD. Rates per minute and proportions of single words and gesture + words had significant correlations with measures of social communication, language skills, and autism symptoms.

CONCLUSIONS

Results suggest that a significantly lower rate per minute of sounds and gesture + sound combinations was a distinguishing feature of ASD in our sample. Further, limited use of single words and gesture + single words was observed in children ASD and DD. Significant prospective relationships between single words and gesture + words with language skills measured over a year later underscores the importance of acquiring these forms.

IMPLICATIONS

Results support the idea that clinicians should include opportunities to observe and encourage coordinated social communication while screening and assessing young children for DD and ASD in the home environment. The significant associations between rate of single words and gesture + word combinations with language development over a year later have implications for incorporating intervention targets that encourage the use of gesture-speech combinations.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

Individual differences in gesture interpretation predict children's propensity to pick a gesturer as a good informant

To learn from others, children rely on cues (e.g., familiarity, confidence) to infer who around them will provide useful information. We extended this research to ask whether children will use an informant's inclination to gesture as a marker of whether or not the informant is a good person to learn from. Children (N = 459, ages 4-12 years) watched short videos in which actresses made statements accompanied by meaningful iconic gestures, beat gestures (which act as prosodic markers with speech), or no gestures. After each trial, children were asked "Who do you think would be a good teacher?" (good teacher [experimental] condition) or "Who do you think would be a good friend?" (good friend [control] condition). Results show that children do believe that someone who produces iconic gesture would make a good teacher compared with someone who does not, but this is only later in childhood and only if children have the propensity to see gesture as meaningful. The same effects were not found in the good friend condition, indicating that children's responses are not just about liking an adult who gestures more. These findings have implications for how children attend to and learn from instructional gesture.

Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources

Speakers use spontaneous hand gestures as they speak and think. These gestures serve many functions for speakers who produce them as well as for listeners who observe them. To date, studies in the gesture literature mostly focused on group-comparisons or the external sources of variation to examine when people use, process, and benefit from using and observing gestures. However, there are also internal sources of variation in gesture use and processing. People differ in how frequently they use gestures, how salient their gestures are, for what purposes they produce gestures, and how much they benefit from using and seeing gestures during comprehension and learning depending on their cognitive dispositions. This review addresses how individual differences in different cognitive skills relate to how people employ gestures in production and comprehension across different ages (from infancy through adulthood to healthy aging) from a functionalist perspective. We conclude that speakers and listeners can use gestures as a compensation tool during communication and thinking that interacts with individuals' cognitive dispositions.

The effects of observing different gestures during storytelling on the recall of path and event information in 5-year-olds and adults

This study examined (a) how observing different types of gestures while listening to a story affected the recall of path and event information in 5-year-old children (n = 71) and adults (n = 55) and (b) whether the effects of gesture type on children's recall of information were related to individual differences such as working memory, language abilities, spontaneous gesture use, and gesture production during the recall task. Participants were asked four questions to measure their spontaneous gesture frequency. They then listened to a story that included different path and event information. Depending on the assigned condition, participants listened to the story with the narrator producing iconic gestures (gestures having semantic meaning), beat gestures (rhythmic hand movements), or no gesture. We then asked participants to relate what happened in the story and administered a recognition task about the story. Children were given standardized tests to assess their language and working memory skills. Children and adults best recalled the story after observing iconic gestures as compared with children and adults presented with beat gestures or no gestures. Children who were exposed to iconic gestures during encoding better recalled event information than children in the other conditions. Children's language abilities, but not working memory, were related to their recall performance. More important, children with better expressive language abilities benefitted more from seeing iconic gestures. These results suggest that observing iconic gestures at encoding facilitates recall and that children's language skills could play a role in encoding and using specific information provided by gestures.

Language and perception: Introduction to the Special Issue “Speakers and Listeners in the Visual World”

Language and perception are two central cognitive systems. Until relatively recently, however, the interaction between them has been examined only partially and not from an over-arching theoretical perspective. Yet it has become clear that linguistic and perceptual interactions are essential to understanding both typical and atypical human behaviour. In this editorial, we examine the link between language and perception across three domains. First, we present a brief review of work investigating the importance of perceptual features, particularly shape bias, when learning names for novel objects—a critical skill acquired during language development. Second, we describe the Visual World Paradigm, an experimental method uniquely suited to investigate the language-perception relationship. Studies using the Visual World Paradigm demonstrate that the relationship between linguistic and perceptual information during processing is both intricate and bi-directional: linguistic cues guide interpretation of visual scenes, while perceptual information shapes interpretation of linguistic input. Finally, we turn to a discussion of co-speech gesture focusing on iconic gestures which depict aspects of the visual world (e.g., motion, shape). The relationship between language and these semantically-meaningful gestures is likewise complex and bi-directional. However, more research is needed to illuminate the exact circumstances under which iconic gestures shape language production and comprehension. In conclusion, although strong evidence exists supporting a critical relationship between linguistic and perceptual systems, the exact levels at which these two systems interact, the time-course of the interaction, and what is driving the interaction, remain largely open questions in need of future research.

Speech-accompanying gestures are not processed by the language-processing mechanisms

Speech-accompanying gestures constitute one information channel during communication. Some have argued that processing gestures engages the brain regions that support language comprehension. However, studies that have been used as evidence for shared mechanisms suffer from one or more of the following limitations: they (a) have not directly compared activations for gesture and language processing in the same study and relied on the fallacious reverse inference (Poldrack, 2006) for interpretation, (b) relied on traditional group analyses, which are bound to overestimate overlap (e.g., Nieto-Castañon and Fedorenko, 2012), (c) failed to directly compare the magnitudes of response (e.g., Chen et al., 2017), and (d) focused on gestures that may have activated the corresponding linguistic representations (e.g., "emblems"). To circumvent these limitations, we used fMRI to examine responses to gesture processing in language regions defined functionally in individual participants (e.g., Fedorenko et al., 2010), including directly comparing effect sizes, and covering a broad range of spontaneously generated co-speech gestures. Whenever speech was present, language regions responded robustly (and to a similar degree regardless of whether the video contained gestures or grooming movements). In contrast, and critically, responses in the language regions were low - at or slightly above the fixation baseline - when silent videos were processed (again, regardless of whether they contained gestures or grooming movements). Brain regions outside of the language network, including some in close proximity to its regions, differentiated between gestures and grooming movements, ruling out the possibility that the gesture/grooming manipulation was too subtle. Behavioral studies on the critical video materials further showed robust differentiation between the gesture and grooming conditions. In summary, contra prior claims, language-processing regions do not respond to co-speech gestures in the absence of speech, suggesting that these regions are selectively driven by linguistic input (e.g., Fedorenko et al., 2011). Although co-speech gestures are uncontroversially important in communication, they appear to be processed in brain regions distinct from those that support language comprehension, similar to other extra-linguistic communicative signals, like facial expressions and prosody.

Unpacking the Ontogeny of Gesture Understanding: How Movement Becomes Meaningful Across Development

Gestures, hand movements that accompany speech, affect children's learning, memory, and thinking (e.g., Goldin-Meadow, 2003). However, it remains unknown how children distinguish gestures from other kinds of actions. In this study, 4- to 9-year-olds (n = 339) and adults (n = 50) described one of three scenes: (a) an actor moving objects, (b) an actor moving her hands in the presence of objects (but not touching them), or (c) an actor moving her hands in the absence of objects. Participants across all ages were equally able to identify actions on objects as goal directed, but the ability to identify empty-handed movements as representational actions (i.e., as gestures) increased with age and was influenced by the presence of objects, especially in older children.