From hands to minds: Gestures promote understanding

Impact of social context on human facial and gestural emotion expressions

Humans flexibly adapt expressions of emotional messages when interacting with others. However, detailed information on how specific parts of the face and hands move in socio-emotional contexts is missing. We identified individual gesture and facial movements (through automated face tracking) of N = 80 participants in the UK, produced while watching amusing, fearful, or neutral movie scenes either alone or with a social partner. Amusing and fearful scenes, more so than neutral scenes, led to an overall increase in facial and gesture movements, confirming emotional responding. Furthermore, social context facilitated movements in the lower instead of upper facial areas, as well as gesture use. These findings highlight emotional signaling components that likely underwent selection for communication, a result we discuss in comparison with the nonhuman primate literature. To facilitate ecologically valid and cross-cultural comparisons on human emotion communication, we additionally offer a new stimuli database of the recorded naturalistic facial expressions.

Bridging Communication Gaps: A Study on Effective Patient Communication Among Respiratory Therapy Students and Interns

Introduction Effective communication in healthcare plays a pivotal role, significantly impacting patient experiences and outcomes. While much of the current literature focuses on communication dynamics among physicians and nurses, a gap exists in understanding these dynamics within allied health professions such as respiratory therapy. This study explores the knowledge, attitudes, and awareness of patient communication among respiratory therapy students and interns. Methods This descriptive cross-sectional study investigated the knowledge, attitudes, and awareness of effective communication methods with patients among respiratory therapy students and interns in Jeddah, Saudi Arabia. Using a validated self-administered questionnaire, the study surveyed 350 individuals from three universities and associated hospitals. Results The analysis involved 350 participants, with females comprising 55.1%. The study found that the highest level of agreement (mean 4.6±0.62) was regarding essential knowledge related to introducing respiratory therapists to patients during communication. Female students demonstrated significant proficiency in concluding patient interviews (P=0.033), while male students excelled in comprehending communication methods with unconscious patients (P=0.010). Interns exhibited the most comprehensive understanding of patient communication skills, particularly in employing open-ended questions (P=0.009) and allowing adequate time for patients to express their concerns (P=0.020). Gender and academic progression were identified as factors influencing patient communication skills in respiratory therapy students and interns. Conclusion This study highlights the need for tailored communication training for respiratory therapy students and interns. It emphasizes the importance of enhancing proficiency in this vital field by addressing knowledge gaps and identifying areas for improvement.

Early or synchronized gestures facilitate speech recall-a study based on motion capture data

Introduction

Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.

Methods

To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.

Results

Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.

Discussion

These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

Major Depression and the Perception of Affective Instrumental and Expressive Gestures: An fMRI Investigation

Major depressive disorder (MDD) is associated with biased perception of human movement. Gesture is important for communication and in this study we investigated neural correlates of gesture perception in MDD. We hypothesised different neural activity between individuals with MDD and typical individuals when viewing instrumental and expressive gestures that were negatively or positively valenced. Differences were expected in brain areas associated with gesture perception, including superior temporal, frontal, and emotion processing regions. We recruited 12 individuals with MDD and 12 typical controls matched on age, gender, and handedness. They viewed gestures displayed by stick figures while functional magnetic resonance imaging (fMRI) was performed. Results of a random effects three-way mixed ANOVA indicated that individuals with MDD had greater activity in the right claustrum compared to controls, regardless of gesture type or valence. Additionally, we observed main effects of gesture type and valence, regardless of group. Perceiving instrumental compared to expressive gestures was associated with greater activity in the left cuneus and left superior temporal gyrus, while perceiving negative compared to positive gestures was associated with greater activity in the right precuneus and right lingual gyrus. We also observed a two-way interaction between gesture type and valence in various brain regions.

HotGestures: Complementing Command Selection and Use with Delimiter-Free Gesture-Based Shortcuts in Virtual Reality

Conventional desktop applications provide users with hotkeys as shortcuts for triggering different functionality. In this paper we consider what constitutes an effective parallel to hotkeys in a 3D interaction space where the input modality is no longer limited to the use of a keyboard. We propose HotGestures: a gesture-based interaction system for rapid tool selection and usage. Hand gestures are frequently used during human communication to convey information and provide natural associations with meaning. HotGestures provide shortcuts for users to seamlessly activate and use virtual tools by performing hand gestures. This approach naturally complements conventional menu interactions. We evaluate the potential of HotGestures in a set of two user studies and observe that our gesture-based technique provides fast and effective shortcuts for tool selection and usage. Participants found HotGestures to be distinctive, fast, and easy to use while also complementing conventional menu-based interaction.

The roles of motion, gesture, and embodied action in the processing of mathematical concepts

This article discusses perspective and frame of reference in the metaphorical description of mathematical concepts in terms of motions, gestures, and embodied actions. When a mathematical concept is described metaphorically in terms of gestures, embodied actions, or fictive motions, the motor system comes into play to ground and understand that concept. Every motion, gesture, or embodied action involves a perspective and a frame of reference. The flexibility in taking perspective and frame of reference allows people to embody a mathematical concept or idea in various ways. Based on the findings of past studies, it is suggested that the graphical representation of a mathematical concept may activate those areas of the motor system that are involved in the production of that graphical representation. This is supported by studies showing that when observers look at a painting or handwritten letters, they simulate the painter's or writer's hand movements during painting or writing. Likewise, the motor system can contribute to the grounding of abstract mathematical concepts, such as functions, numbers, and arithmetic operations.

Drawing as a Space for Social-Cognitive Interaction

Drawing is recognized as a powerful tool to learn science. Although current research has enriched our understanding of the potential of learning through drawing, scarce attention has been given to the social-cognitive interactions that occur when students jointly create drawings to understand and explain phenomena in science. This article is based on the distributed and embodied cognition theories and it adopted the notion of we-space, defined as a complex social-cognitive space, dynamically established and managed during the ongoing interactions of the individuals, when they manipulate and exploit a shared space. The goal of the study was to explore the role that collaborative drawing plays in shaping the social-cognitive interaction among students. We examine this by a fine-grain multimodal analysis of a pair of middle school students, who jointly attempted to understand and explain a chemical phenomenon by creating drawings and thinking with them. Our findings suggest that collaborative drawing played a key role in (i) establishing a genuine shared-action space, a we-space, and that within this we-space it had two major functions: (ii) enabling collective thinking-in-action and (iii) simplifying communication. We argue that drawing, as a joint activity, has a potential for learning, not restricted to the cognitive process related to the activity of creating external visual representations on paper; instead, the benefits of drawing lie in action in space. Creating these representations is more than a process of externalization of thought: it is part of a process of collective thinking-in-action.

Thinking Tools: Gestures Change Thought About Time

Our earliest tools are our bodies. Our hands raise and turn and toss and carry and push and pull, our legs walk and climb and kick allowing us to move and act in the world and to create the multitude of artifacts that improve our lives. The list of actions made by our hands and feet and other parts of our bodies is long. What is more remarkable is we turn those actions in the world into actions on thought through gestures, language, and graphics, thereby creating cognitive tools that expand the mind. The focus here is gesture; gestures transform actions on perceptible objects to actions on imagined thoughts, carrying meaning with them rapidly, precisely, and directly. We review evidence showing that gestures enhance our own thinking and change the thought of others. We illustrate the power of gestures in studies showing that gestures uniquely change conceptions of time, from sequential to simultaneous, from sequential to cyclical, and from a perspective embedded in a timeline to an external perspective looking on a timeline, and by so doing obviate the ambiguities of an embedded perspective. We draw parallels between representations in gesture and in graphics; both use marks or actions arrayed in space to communicate more immediately than symbolic language.

Removing Hand Form Information Specifically Impairs Emotion Recognition for Fearful and Angry Body Stimuli

Emotion perception research has largely been dominated by work on facial expressions, but emotion is also strongly conveyed from the body. Research exploring emotion recognition from the body tends to refer to “the body” as a whole entity. However, the body is made up of different components (hands, arms, trunk, etc.), all of which could be differentially contributing to emotion recognition. We know that the hands can help to convey actions and, in particular, are important for social communication through gestures, but we currently do not know to what extent the hands influence emotion recognition from the body. Here, 93 adults viewed static emotional body stimuli with either the hands, arms, or both components removed and completed a forced-choice emotion recognition task. Removing the hands significantly reduced recognition accuracy for fear and anger but made no significant difference to the recognition of happiness and sadness. Removing the arms had no effect on emotion recognition accuracy compared with the full-body stimuli. These results suggest the hands may play a key role in the recognition of emotions from the body.

Embodied science and mixed reality: How gesture and motion capture affect physics education

A mixed design was created using text and game-like multimedia to instruct in the content of physics. The study assessed which variables predicted learning gains after a 1-h lesson on the electric field. The three manipulated variables were: (1) level of embodiment; (2) level of active generativity; and (3) presence of story narrative. Two types of tests were administered: (1) a traditional text-based physics test answered with a keyboard; and (2) a more embodied, transfer test using the Wacom large tablet where learners could use gestures (long swipes) to create vectors and answers. The 166 participants were randomly assigned to four conditions: (1) symbols and text; (2) low embodied; (3) high embodied/active; or (4) high embodied/active with narrative. The last two conditions were active because the on-screen content could be manipulated with gross body gestures gathered via the Kinect sensor. Results demonstrated that the three groups that included embodiment learned significantly more than the symbols and text group on the traditional keyboard post-test. When knowledge was assessed with the Wacom tablet format that facilitated gestures, the two active gesture-based groups scored significantly higher. In addition, engagement scores were significantly higher for the two active embodied groups. The Wacom results suggest test sensitivity issues; the more embodied test revealed greater gains in learning for the more embodied conditions. We recommend that as more embodied learning comes to the fore, more sensitive tests that incorporate gesture be used to accurately assess learning. The predicted differences in engagement and learning for the condition with the graphically rich story narrative were not supported. We hypothesize that a narrative effect for motivation and learning may be difficult to uncover in a lab experiment where participants are primarily motivated by course credit. Several design principles for mediated and embodied science education are proposed.