Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

How and Why People Synchronize: An Integrated Perspective

Academic AbstractInterpersonal synchrony, the alignment of behavior and/or physiology during interactions, is a pervasive phenomenon observed in diverse social contexts. Here we synthesize across contexts and behaviors to classify the different forms and functions of synchrony. We provide a concise framework for classifying the manifold forms of synchrony along six dimensions: periodicity, discreteness, spatial similarity, directionality, leader-follower dynamics, and observability. We also distill the various proposed functions of interpersonal synchrony into four interconnected functions: reducing complexity and improving understanding, accomplishing joint tasks, strengthening social connection, and influencing partners' behavior. These functions derive from first principles, emerge from each other, and are accomplished by some forms of synchrony more than others. Effective synchrony flexibly adapts to social goals and more synchrony is not always better. Our synthesis offers a shared framework and language for the field, allowing for better cross-context and cross-behavior comparisons, generating new hypotheses, and highlighting future research directions.

Classifying interpersonal synchronization states using a data-driven approach: implications for social interaction understanding

This study presents a data-driven approach to identifying interpersonal motor synchrony states by analyzing hand movements captured from a 3D depth camera. Utilizing a single frame from the experiment, an XGBoost machine learning model was employed to differentiate between spontaneous and intentional synchrony modes with nearly [Formula: see text] accuracy. Our findings demonstrate a consistent pattern across subjects, revealing that movement velocity tends to be slower in synchrony modes. These insights support the notion that the relationship between velocity and synchrony is influenced by the cognitive load required for the task, with slower movements leading to higher synchrony in tasks demanding higher cognitive load. This work not only contributes to the limited literature on algorithms for identifying interpersonal synchrony but also has potential implications for developing new metrics to assess real-time human social interactions, understanding social interaction, and diagnosing and developing treatment strategies for social deficits associated with conditions such as Autism Spectrum Disorder.

Identifying Signatures of Perceived Interpersonal Synchrony

Interpersonal synchrony serves as a subtle, yet powerful bonding mechanism in social interactions. Problematically, the term ‘synchrony’ has been used to label a variety of distinct aspects of interpersonal coordination, such as postural similarities or movement activity entrainment. Accordingly, different algorithms have been suggested to quantify interpersonal synchrony. Yet, it remains unknown whether the different measures of synchrony represent correlated features of the same perceivable core phenomenon. The current study addresses this by comparing the suitability of a set of algorithms with respect to their association with observers’ judgments of dyadic synchrony and leader-followership. One-hundred fifteen observers viewed computer animations of characters portraying the movements of real dyads who performed a repetitive motor task with instruction to move in unison. Animations were based on full-body motion capture data synchronously collected for both partners during the joint exercise. Results showed most synchrony measures significantly correlated with (a) perceived synchrony and (b) the perceived level of balance of leading/following by each dyad member. Phase synchrony and Pearson correlations were associated most strongly with the observer ratings. This might be typical for intentional, structured forms synchrony such as ritualized group activities. It remains open if these findings also apply to spontaneous forms of synchrony as, for instance, occurring in free-running conversations.

Collective Rhythm as an Emergent Property During Human Social Coordination

The literature on social interactions has shown that participants coordinate not only at the behavioral but also at the physiological and neural levels, and that this coordination gives a temporal structure to the individual and social dynamics. However, it has not been fully explored whether such temporal patterns emerge during interpersonal coordination beyond dyads, whether this phenomenon arises from complex cognitive mechanisms or from relatively simple rules of behavior, or which are the sociocultural processes that underlie this phenomenon. We review the evidence for the existence of group-level rhythmic patterns that result from social interactions and argue that the complexity of group dynamics can lead to temporal regularities that cannot be predicted from the individual periodicities: an emergent collective rhythm. Moreover, we use this interpretation of the literature to discuss how taking into account the sociocultural niche in which individuals develop can help explain the seemingly divergent results that have been reported on the social influences and consequences of interpersonal coordination. We make recommendations on further research to test these arguments and their relationship to the feeling of belonging and assimilation experienced during group dynamics.

Suggestions for Improving the Investigation of Gesture in Aphasia

Purpose When we speak, we gesture, and indeed, persons with aphasia gesture more frequently. The reason(s) for this is still being investigated, spurring an increase in the number of studies of gesture in persons with aphasia. As the number of studies increases, so too does the need for a shared set of best practices for gesture research in aphasia. After briefly reviewing the importance and use of gesture in persons with aphasia, this viewpoint puts forth methodological and design considerations when evaluating gesture in persons with aphasia. Method & Results We explore several different design and methodological considerations for gesture research specific to persons with aphasia, such as video angle specifications, data collection techniques, and analysis considerations. The goal of these suggestions is to develop transparent and reproducible methods for evaluating gesture in aphasia to build a solid foundation for continued work in this area. Conclusions We have proposed that it is critical to evaluate multimodal communication in a methodologically robust way to facilitate increased knowledge about the relationship of gesture to spoken language, cognition, and to other aspects of living with aphasia and recovery from aphasia. We conclude by postulating future directions for gesture research in aphasia.

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

Silent gestures consist of complex multi‐articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content.

Alignment in Multimodal Interaction: An Integrative Framework

When people are engaged in social interaction, they can repeat aspects of each other's communicative behavior, such as words or gestures. This kind of behavioral alignment has been studied across a wide range of disciplines and has been accounted for by diverging theories. In this paper, we review various operationalizations of lexical and gestural alignment. We reveal that scholars have fundamentally different takes on when and how behavior is considered to be aligned, which makes it difficult to compare findings and draw uniform conclusions. Furthermore, we show that scholars tend to focus on one particular dimension of alignment (traditionally, whether two instances of behavior overlap in form), while other dimensions remain understudied. This hampers theory testing and building, which requires a well-defined account of the factors that are central to or might enhance alignment. To capture the complex nature of alignment, we identify five key dimensions to formalize the relationship between any pair of behavior: time, sequence, meaning, form, and modality. We show how assumptions regarding the underlying mechanism of alignment (placed along the continuum of priming vs. grounding) pattern together with operationalizations in terms of the five dimensions. This integrative framework can help researchers in the field of alignment and related phenomena (including behavior matching, mimicry, entrainment, and accommodation) to formulate their hypotheses and operationalizations in a more transparent and systematic manner. The framework also enables us to discover unexplored research avenues and derive new hypotheses regarding alignment.

Introducing the NEMO-Lowlands iconic gesture dataset, collected through a gameful human-robot interaction

This paper describes a novel dataset of iconic gestures, together with a publicly available robot-based elicitation method to record these gestures, which consists of playing a game of charades with a humanoid robot. The game was deployed at a science museum (NEMO) and a large popular music festival (Lowlands) in the Netherlands. This resulted in recordings of 428 participants, both adults and children, performing 3715 silent iconic gestures for 35 different objects in a naturalistic setting. Our dataset adds to existing collections of iconic gesture recordings in two important ways. First, participants were free to choose how they represented the broad concepts using gestures, and they were asked to perform a second attempt if the robot did not recognize their gesture the first time. This provides insight into potential repair strategies that might be used. Second, by making the interactive game available we enable other researchers to collect additional recordings, for different concepts, and in diverse cultures or contexts. This can be done in a consistent manner because a robot is used as a confederate in the elicitation procedure, which ensures that every data collection session plays out in the same way. The current dataset can be used for research into human gesturing behavior, and as input for the gesture recognition and production capabilities of robots and virtual agents.