Effects of Encouraging the Use of Gestures on Speech

Development and Psychometric Examination of a New Social Competence Outcome Measure for Children with Autism Spectrum Disorder: The Observational Social Competence Assessment

Current assessments of social competence for children with autism spectrum disorder (ASD) are mostly designed for screening or diagnosis, not for measuring outcomes. This study aimed to develop a professional-administrated outcome measure, the Observational Social Competence Assessment (OSCA), and examine its psychometric properties. The OSCA was constructed based on a multidimensional view of social competence (i.e., social skill elements, social reciprocity, and social adjustment). For psychometric evaluation, 89 children with ASD between 3 and 12 years (mean = 70.69 months, SD = 15.31) were assessed with the OSCA and with assessments of ASD symptoms, verbal comprehension ability, and adaptive function. The results show that the OSCA has good internal consistency (Cronbach's α = 0.820-0.954), test-retest reliability (intraclass correlation coefficients [ICC] = 0.917-0.960), and inter-rater reliability (ICC = 0.905-0.974). The OSCA also has good convergent (r = 0.508-0.703, p < 0.01) and divergent validity (r = 0.105, p = 0.496), as well as good responsiveness to changes in the social adjustment dimension (Cohen's d = 1.26 and standardized response mean [SRM] = 1.92). Conclusively, these results show that the OSCA is sufficiently reliable, valid and responsive to be applied as an outcome measure of social competence in children with ASD.

Give your ideas a hand: the role of iconic hand gestures in enhancing divergent creative thinking

Hand gestures play an integral role in multimodal language and communication. Even though the self-oriented functions of gestures, such as activating a speaker's lexicon and maintaining visuospatial imagery, have been emphasized, gestures' functions in creative thinking are not well-established. In the current study, we investigated the role of iconic gestures in verbal divergent thinking-a creative thinking process related to generating many novel ideas. Based on previous findings, we hypothesized that iconic gesture use would facilitate divergent thinking in young adults, especially those with high mental imagery skills. Participants performed Guildford's Alternative Uses Task in a gesture-spontaneous and in a gesture-encouraged condition. We measured fluency (number of ideas), originality (uniqueness of ideas), flexibility (number of idea categories), and elaboration (number of details) in divergent thinking. The results showed that producing iconic gestures in the gesture-encouraged condition positively predicted fluency, originality, and elaboration. In the gesture-spontaneous condition, producing iconic gestures also positively predicted elaboration but negatively predicted flexibility. Mental imagery skills did not interact with the effects of gestures on divergent thinking. These results suggest that iconic gestures are a promising candidate for enhancing almost all aspects of divergent thinking. Overall, the current study adds a new dimension to the self-oriented function of iconic gestures, that is, their contribution to creative thinking.

Origins Of Vocal-Entangled Gesture

Gestures during speaking are typically understood in a representational framework: they represent absent or distal states of affairs by means of pointing, resemblance, or symbolic replacement. However, humans also gesture along with the rhythm of speaking, which is amenable to a non-representational perspective. Such a perspective centers on the phenomenon of vocal-entangled gestures and builds on evidence showing that when an upper limb with a certain mass decelerates/accelerates sufficiently, it yields impulses on the body that cascade in various ways into the respiratory-vocal system. It entails a physical entanglement between body motions, respiration, and vocal activities. It is shown that vocal-entangled gestures are realized in infant vocal-motor babbling before any representational use of gesture develops. Similarly, an overview is given of vocal-entangled processes in non-human animals. They can frequently be found in rats, bats, birds, and a range of other species that developed even earlier in the phylogenetic tree. Thus, the origins of human gesture lie in biomechanics, emerging early in ontogeny and running deep in phylogeny.

The Design and Observed Effects of Robot-Performed Manual Gestures: A Systematic Review

Communication using manual (hand) gestures is considered a defining property of social robots, and their physical embodiment and presence, therefore we see a need for a comprehensive overview of the state of the art in social robots that use gestures. This systematic literature review aims to address this need by (1) describing the gesture production process of a social robot, including the design and planning steps, and (2) providing a survey of the effects of robot-performed gestures on human-robot interactions in a multitude of domains. We identify patterns and themes from the existing body of literature, resulting in nine outstanding questions for research on robot-performed gestures regarding: developments in sensor technology and AI, structuring the gesture design and evaluation process, the relationship between physical appearance and gestures, the effects of planning on the overall interaction, standardizing measurements of gesture ‘quality’, individual differences, gesture mirroring, whether human-likeness is desirable, and universal accessibility of robots. We also reflect on current methodological practices in studies of robot-performed gestures, and suggest improvements regarding replicability, external validity, measurement instruments used, and connections with other disciplines. These outstanding questions and methodological suggestions can guide future work in this field of research.

Using key-word signing to support learners in South African schools: a study of teachers' perceptions

Key-word-signing (KWS), an unaided augmentative and alternative communication (AAC) strategy, has an important role to play in the education system. To date, limited research exists regarding the use of unaided AAC strategies in the school environment, especially in low- and middle-income countries. To address this research gap, the current study aimed to explore the perceptions of South African teachers toward the use of KWS strategies in the educational context. The study used a descriptive paper-based survey comprised of both closed and open-ended questions that was completed by 101 special education teachers. The sample was drawn from 10 public and independent schools for learners with special education needs in five urban school districts within South Africa's Gauteng province. These schools accommodate students with a broad spectrum of disabilities and are likely to include learners with complex communication needs. The participants identified various factors and challenges that influenced the use of KWS within the education system (e.g., school culture, parents and other team members, community awareness, learner-related considerations, and training needs). These factors and challenges are assessed and discussed in relation to both local and international norms. Recommendations are made regarding expanding both an understanding and acceptance of unaided AAC strategies as a further means to support learning in the educational context.

Semi-automation of gesture annotation by machine learning and human collaboration

Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

Speakers exhibit a multimodal Lombard effect in noise

In everyday conversation, we are often challenged with communicating in non-ideal settings, such as in noise. Increased speech intensity and larger mouth movements are used to overcome noise in constrained settings (the Lombard effect). How we adapt to noise in face-to-face interaction, the natural environment of human language use, where manual gestures are ubiquitous, is currently unknown. We asked Dutch adults to wear headphones with varying levels of multi-talker babble while attempting to communicate action verbs to one another. Using quantitative motion capture and acoustic analyses, we found that (1) noise is associated with increased speech intensity and enhanced gesture kinematics and mouth movements, and (2) acoustic modulation only occurs when gestures are not present, while kinematic modulation occurs regardless of co-occurring speech. Thus, in face-to-face encounters the Lombard effect is not constrained to speech but is a multimodal phenomenon where the visual channel carries most of the communicative burden.

Gesture-speech physics in fluent speech and rhythmic upper limb movements

It is commonly understood that hand gesture and speech coordination in humans is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech vocalization has been studied in steady-state vocalization and monosyllabic utterances, where forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (n = 37), we extend this previous line of work to show that gesture-speech physics also impacts fluent speech. Compared with nonmovement, participants who are producing fluent self-formulated speech while rhythmically moving their limbs demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lower-impulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.

Introducing the NEMO-Lowlands iconic gesture dataset, collected through a gameful human-robot interaction

This paper describes a novel dataset of iconic gestures, together with a publicly available robot-based elicitation method to record these gestures, which consists of playing a game of charades with a humanoid robot. The game was deployed at a science museum (NEMO) and a large popular music festival (Lowlands) in the Netherlands. This resulted in recordings of 428 participants, both adults and children, performing 3715 silent iconic gestures for 35 different objects in a naturalistic setting. Our dataset adds to existing collections of iconic gesture recordings in two important ways. First, participants were free to choose how they represented the broad concepts using gestures, and they were asked to perform a second attempt if the robot did not recognize their gesture the first time. This provides insight into potential repair strategies that might be used. Second, by making the interactive game available we enable other researchers to collect additional recordings, for different concepts, and in diverse cultures or contexts. This can be done in a consistent manner because a robot is used as a confederate in the elicitation procedure, which ensures that every data collection session plays out in the same way. The current dataset can be used for research into human gesturing behavior, and as input for the gesture recognition and production capabilities of robots and virtual agents.

Beat Gestures for Comprehension and Recall: Differential Effects of Language Learners and Native Listeners

Previous work has shown how native listeners benefit from observing iconic gestures during speech comprehension tasks of both degraded and non-degraded speech. By contrast, effects of the use of gestures in non-native listener populations are less clear and studies have mostly involved iconic gestures. The current study aims to complement these findings by testing the potential beneficial effects of beat gestures (non-referential gestures which are often used for information- and discourse marking) on language recall and discourse comprehension using a narrative-drawing task carried out by native and non-native listeners. Using a within-subject design, 51 French intermediate learners of English participated in a narrative-drawing task. Each participant was assigned 8 videos to watch, where a native speaker describes the events of a short comic strip. Videos were presented in random order, in four conditions: in Native listening conditions with frequent, naturally-modeled beat gestures, in Native listening conditions without any gesture, in Non-native listening conditions with frequent, naturally-modeled beat gestures, and in Non-native listening conditions without any gesture. Participants watched each video twice and then immediately recreated the comic strip through their own drawings. Participants' drawings were then evaluated for discourse comprehension (via their ability to convey the main goals of the narrative through their drawings) and recall (via the number of gesturally-marked elements in the narration that were included in their drawings). Results showed that for native listeners, beat gestures had no significant effect on either recall or comprehension. In non-native speech, however, beat gestures led to significantly lower comprehension and recall scores. These results suggest that frequent, naturally-modeled beat gestures in longer discourses may increase cognitive load for language learners, resulting in negative effects on both memory and language understanding. These findings add to the growing body of literature that suggests that gesture benefits are not a "one-size-fits-all" solution, but rather may be contingent on factors such as language proficiency and gesture rate, particularly in that whenever beat gestures are repeatedly used in discourse, they inherently lose their saliency as markers of important information.

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

Expressive moments in communicative hand gestures often align with emphatic stress in speech. It has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impulses on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) repeatedly uttered consonant-vowel (/pa/) mono-syllables while moving in particular phase relations with speech, or not moving the upper limbs. This study shows that respiration-related activity is affected by (especially high-impulse) gesturing when vocalizations occur near peaks in physical impulse. This study further shows that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, tight relations between respiration-related activity and vocalization were observed, even in the absence of movement, but even more so when upper-limb movement is present. The current findings expand a developing line of research showing that speech is modulated by functional biomechanical linkages between hand gestures and the respiratory system. This identification of gesture-speech biomechanics promises to provide an alternative phylogenetic, ontogenetic, and mechanistic explanatory route of why communicative upper limb movements co-occur with speech in humans.