Combining Recurrence Analysis and Automatic Movement Extraction from Video Recordings to Study Behavioral Coupling in Face-to-Face Parent-Child Interactions

The Dynamical Hypothesis in Situ: Challenges and Opportunities for a Dynamical Social Approach to Interpersonal Coordination

Over the past three decades, Van Gelder's dynamical hypothesis has been instrumental in reconceptualizing the ways in which perception-action-cognition unfolds over time and in context. Here, I examine how the dynamical approach has enriched the theoretical understanding of social dynamics within cognitive science, with a particular focus on interpersonal coordination. I frame this review around seven principles in dynamical systems: three that are well-represented in interpersonal coordination research to date (emergent behavior, context-sensitive behavior, and attractors) and four that could be useful opportunities for future growth (hysteresis, sensitivity to initial conditions, equifinality, and reciprocal compensation). In addition to identifying specific promising lines of theoretical inquiry, I focus on the significant potential afforded by computationally intensive science-especially in naturally occurring data or trace data. Building on the foundation laid over the past three decades, I argue that looking to increasingly situated and naturalistic settings (and data) is not only necessary to realize the full commitment to the dynamical hypothesis but is also critical to building parsimonious and principled theories of social phenomena.

The dynamics of dyadic interactions between people of different ethnicities depend on their identification with all humanity

Individuals who have the disposition to identify with all humanity declare feeling close to people all over the world, caring about them, and perceiving them as an ingroup. However, never before were such declarations verified by measures of intergroup attitudes less direct than questionnaires, such as approach/avoidance tendencies or dynamical systems methods. Since individuals with higher dispositional identification with all humanity (IWAH) perceive people all over the world as ingroup members, we expected differences in the dynamic of inter-ethnic interactions (spatial distance, coordination, coupling, and leading), depending on a participant's level of IWAH. 227 participants fulfilled the IWAH scale, and those with the highest and lowest scores took part in a laboratory study, performing a task in inter-ethnic dyads. For the first time, an approach that combines a state-of-the-art tracking algorithm with a dynamical systems method was applied in such a context. Our results showed that those higher in IWAH kept a smaller distance from a partner, took a more leading role, and showed better coordination than those lower in IWAH. We found a similar trend for coupling. The results demonstrated the importance of IWAH for inter-ethnic relations and how it may shape non-verbal behaviors. Limitations are discussed.

Social buffering diminishes fear response but does not equal improved fear extinction

Social support during exposure-based psychotherapy is believed to diminish fear and improve therapy outcomes. However, some clinical trials challenge that notion. Underlying mechanisms remain unknown, hindering the understanding of benefits and pitfalls of such approach. To study social buffering during fear extinction, we developed a behavioral model in which partner's presence decreases response to fear-associated stimuli. To identify the neuronal background of this phenomenon, we combined behavioral testing with c-Fos mapping, optogenetics, and chemogenetics. We found that the presence of a partner during fear extinction training causes robust inhibition of freezing; the effect, however, disappears in subjects tested individually on the following day. It is accompanied by lowered activation of the prelimbic (PL) and anterior cingulate (ACC) but not infralimbic (IL) cortex. Accordingly, blocking of IL activity left social buffering intact. Similarly, inhibition of the ventral hippocampus-PL pathway, suppressing fear response after prolonged extinction training, did not diminish the effect. In contrast, inhibition of the ACC-central amygdala pathway, modulating social behavior, blocked social buffering. By reporting that social modulation of fear inhibition is transient and insensitive to manipulation of the fear extinction-related circuits, we show that the mechanisms underlying social buffering during extinction are different from those of individual extinction.

Co-regulation of movements during infant feeding

The process by which infants move from liquid feeding to caregiver-assisted spoon feeding of semi-solid food is quite a dramatic transition. In previous studies, we observed that in the weeks after the introduction to solid food, mother-infant dyads showed increased co-regulation and synchronization of their respective feeding behaviors (e.g. offering food, accepting/refusing, timing). Learning this new way of feeding and eating requires that infants coordinate their position and movements with the complementary position and movements of the caregiver. The present study augments the category-based analysis of this co-regulation by the analysis of coupling in the dyads based on automatically extracted movement data. Previously collected video data from 10 mother-infant dyads were re-analyzed for the purpose of this study. Movement trajectories of mother's hand and infant's face were obtained by applying an automatic movement detection algorithm (TLD, Kalal et al., 2012; for applications to mother-infant interactions see López Pérez et al., 2017). Coordination was assessed by the method of Diagonal Cross Recurrence Profiles (DCRP), which expresses the degree of synchronization at different time lags. Profiles for each dyad from two different occasions --with one visit in the first week of solid feeding and one approximately 4-5 weeks later-- were compared. The results showed that, on average, most synchronization occurred in the first visit at lag 0. In the second visit there was an average delay in synchronization of about 1 s, with leading behavior starting from the infant. This suggests that the coordination was initially closely synchronized and became somewhat looser over time. Possibly, infants have begun to anticipate and guide the feeding movements enacted by the mother. However, our findings underline the idiosyncratic and complex nature of co-regulation of movements during the introduction of solid food. Whereas some dyads showed signs of increased organization, others seemed to disorganize, re-organize, or showed no organization at all. Many (interacting) factors --both individual and contextual-- may be responsible for the observed differences between dyads. Further research is needed to understand why specific synchronization pathways emerge and whether and how these might relate both to later feeding and eating and to the emergent patterns of participation.

Assessing Sensorimotor Synchronisation in Toddlers Using the Lookit Online Experiment Platform and Automated Movement Extraction

Adapting gross motor movement to match the tempo of auditory rhythmic stimulation (sensorimotor synchronisation; SMS) is a complex skill with a long developmental trajectory. Drumming tasks have previously been employed with infants and young children to measure the emergence of rhythmic entrainment, and may provide a tool for identification of those with atypical rhythm perception and production. Here we describe a new protocol for measuring infant rhythmic movement that can be employed at scale. In the current study, 50 two-year-olds drummed along with the audiovisual presentation of four steady rhythms, using videos of isochronous drumming at 400, 500, 600, and 700 ms IOI, and provided their spontaneous motor tempo (SMT) by drumming in silence. Toddlers’ drumming is observed from video recordings made in participants’ own homes, obtained via the Lookit platform for online infant studies. We use OpenPose deep-learning model to generate wireframe estimates of hand and body location for each video. The vertical displacement of the hand was extracted, and the power and frequency of infants’ rhythmic entrainment quantified using Fast Fourier Transforms. We find evidence for age-appropriate tempo-flexibility in our sample. Our results demonstrate the feasibility of a fully digital approach to measuring rhythmic entrainment from within the participant’s home, from early in development.

Dynamics of Remote Communication: Movement Coordination in Video-Mediated and Face-to-Face Conversations

The present pandemic forced our daily interactions to move into the virtual world. People had to adapt to new communication media that afford different ways of interaction. Remote communication decreases the availability and salience of some cues but also may enable and highlight others. Importantly, basic movement dynamics, which are crucial for any interaction as they are responsible for the informational and affective coupling, are affected. It is therefore essential to discover exactly how these dynamics change. In this exploratory study of six interacting dyads we use traditional variability measures and cross recurrence quantification analysis to compare the movement coordination dynamics in quasi-natural dialogues in four situations: (1) remote video-mediated conversations with a self-view mirror image present, (2) remote video-mediated conversations without a self-view, (3) face-to-face conversations with a self-view, and (4) face-to-face conversations without a self-view. We discovered that in remote interactions movements pertaining to communicative gestures were exaggerated, while the stability of interpersonal coordination was greatly decreased. The presence of the self-view image made the gestures less exaggerated, but did not affect the coordination. The dynamical analyses are helpful in understanding the interaction processes and may be useful in explaining phenomena connected with video-mediated communication, such as “Zoom fatigue”.

Changes in the Complexity of Limb Movements during the First Year of Life across Different Tasks

Infants’ limb movements evolve from disorganized to more selectively coordinated during the first year of life as they learn to navigate and interact with an ever-changing environment more efficiently. However, how these coordination patterns change during the first year of life and across different contexts is unknown. Here, we used wearable motion trackers to study the developmental changes in the complexity of limb movements (arms and legs) at 4, 6, 9 and 12 months of age in two different tasks: rhythmic rattle-shaking and free play. We applied Multidimensional Recurrence Quantification Analysis (MdRQA) to capture the nonlinear changes in infants’ limb complexity. We show that the MdRQA parameters (entropy, recurrence rate and mean line) are task-dependent only at 9 and 12 months of age, with higher values in rattle-shaking than free play. Since rattle-shaking elicits more stable and repetitive limb movements than the free exploration of multiple objects, we interpret our data as reflecting an increase in infants’ motor control that allows for stable body positioning and easier execution of limb movements. Infants’ motor system becomes more stable and flexible with age, allowing for flexible adaptation of behaviors to task demands.

Distinct circuits in rat central amygdala for defensive behaviors evoked by socially signaled imminent versus remote danger

Animals display a rich repertoire of defensive responses adequate to the threat proximity. In social species, these reactions can be additionally influenced by the behavior of fearful conspecifics. However, the majority of neuroscientific studies on socially triggered defensive responses focuses on one type of behavior, freezing. To study a broader range of socially triggered reactions and underlying mechanisms, we directly compared two experimental paradigms, mimicking occurrence of the imminent versus remote threat. Observation of a partner currently experiencing aversive stimulation evokes passive defensive responses in the observer rats. Similar interaction with a partner that has just undergone the aversive stimulation prompts animals to increase active exploration. Although the observers display behaviors similar to those of the aversively stimulated demonstrators, their reactions are not synchronized in time, suggesting that observers' responses are caused by the change in their affective state rather than mimicry. Using opsins targeted to behaviorally activated neurons, we tagged central amygdala (CeA) cells implicated in observers' responses to either imminent or remote threat and reactivated them during the exploration of a novel environment. The manipulation revealed that the two populations of CeA cells promote passive or active defensive responses, respectively. Further experiments confirmed that the two populations of cells at least partially differ in expression of molecular markers (protein kinase C-δ [PKC-δ] and corticotropin-releasing factor [CRF]) and connectivity patterns (receiving input from the basolateral amygdala or from the anterior insula). The results are consistent with the literature on single subjects' fear conditioning, suggesting that similar neuronal circuits control defensive responses in social and non-social contexts.

"Dancing" Together: Infant-Mother Locomotor Synchrony

Pre-mobile infants and caregivers spontaneously engage in a sequence of contingent facial expressions and vocalizations that researchers have referred to as a social "dance." Does this dance continue when both partners are free to move across the floor? Locomotor synchrony was assessed in 13- to 19-month-old infant-mother dyads (N = 30) by tracking each partner's step-to-step location during free play. Although infants moved more than mothers, dyads spontaneously synchronized their locomotor activity. For 27 dyads, the spatiotemporal path of one partner uniquely identified the path of the other. Clustering analyses revealed two patterns of synchrony (mother-follow and yo-yo), and infants were more likely than mothers to lead the dance. Like face-to-face synchrony, locomotor synchrony scaffolds infants' interactions with the outside world.

Parent-child interaction during the first year of life in infants at elevated likelihood of autism spectrum disorder

Autism spectrum disorder (ASD) likely emerges from a complex interaction between pre-existing neurodevelopmental vulnerabilities and the environment. The interaction with parents forms a key aspect of an infant's social environment, but few prospective studies of infants at elevated likelihood (EL) for ASD (who have an older sibling with ASD) have examined parent-child interactions in the first year of life. As part of a European multisite network, parent-child dyads of free play were observed at 5 months (62 EL infants, 47 infants at typical likelihood (TL)) and 10 months (101 EL siblings, 77 TL siblings). The newly-developed Parent-Infant/Toddler Coding of Interaction (PInTCI) scheme was used, focusing on global characteristics of infant and parent behaviors. Coders were blind to participant information. Linear mixed model analyses showed no significant group differences in infant or parent behaviors at 5 or 10 months of age (all ps≥0.09, d≤0.36), controlling for infant's sex and age, and parental educational level. However, without adjustments, EL infants showed fewer and less clear initiations at 10 months than TL infants (p = 0.02, d = 0.44), but statistical significance was lost after controlling for parental education (p = 0.09, d = 0.36), which tended to be lower in the EL group. Consistent with previous literature focusing on parent-infant dyads, our findings suggest that differences between EL and TL dyads may only be subtle during the first year of life. We discuss possible explanations and implications for future developmental studies.

Finding Structure in Time: Visualizing and Analyzing Behavioral Time Series

The temporal structure of behavior contains a rich source of information about its dynamic organization, origins, and development. Today, advances in sensing and data storage allow researchers to collect multiple dimensions of behavioral data at a fine temporal scale both in and out of the laboratory, leading to the curation of massive multimodal corpora of behavior. However, along with these new opportunities come new challenges. Theories are often underspecified as to the exact nature of these unfolding interactions, and psychologists have limited ready-to-use methods and training for quantifying structures and patterns in behavioral time series. In this paper, we will introduce four techniques to interpret and analyze high-density multi-modal behavior data, namely, to: (1) visualize the raw time series, (2) describe the overall distributional structure of temporal events (Burstiness calculation), (3) characterize the non-linear dynamics over multiple timescales with Chromatic and Anisotropic Cross-Recurrence Quantification Analysis (CRQA), (4) and quantify the directional relations among a set of interdependent multimodal behavioral variables with Granger Causality. Each technique is introduced in a module with conceptual background, sample data drawn from empirical studies and ready-to-use Matlab scripts. The code modules showcase each technique's application with detailed documentation to allow more advanced users to adapt them to their own datasets. Additionally, to make our modules more accessible to beginner programmers, we provide a "Programming Basics" module that introduces common functions for working with behavioral timeseries data in Matlab. Together, the materials provide a practical introduction to a range of analyses that psychologists can use to discover temporal structure in high-density behavioral data.

What Do Young Infants Do During Eye-Tracking Experiments? IP-BET - A Coding Scheme for Quantifying Spontaneous Infant and Parent Behaviour

Eye-tracking measurement of looking is the fundamental method in infancy research. Over the last few decades it has provided many of the most significant discoveries in developmental psychology. Infants engage in looking tasks and use their bodies for learning differently from adults, yet, the breadth of their behavioural repertoire and the constraints that the testing situation places on them remain under-explored. Young infants are tested in close physical proximity to their parent, interact during the experiment and rely on the parent to stay engaged in the task. Infants may also engage a different set of skills (e.g. when self-regulating) to perform the very same looking tasks in comparison with adult participants. We devised a coding scheme to systematically analyse task-relevant (attention to the screen) and extraneous behaviours [body movement, self-touch, non-nutritive sucking (NNS), affect] that infants exhibit during an eye-tracking session. We also measured parental behaviours (attention to the screen or the child), including dyadic interactions with the infant (talking, physical contact). We outline the rationale for the scheme and present descriptive data on the behaviour of a large group of typical 5- and 6-month-olds (n = 94) during three standard eye-tracking tasks in two seating arrangements. The majority of infants showed very high and consistent within-group attention to the screen, while there were large individual differences in the amount of limb and body movement and the use of self-regulatory behaviours (NNS, self-touch, object manipulation). Very few sex differences were found. Parents spent most time attending to the screen, but engaged in some forms of dyadic interaction, despite being given standard task instructions that minimise parental interference. Our results demonstrate the variability in infants' extraneous behaviours during standard eye-tracking despite comparable duration of attention to the screen. They show that spontaneous interactions with the parent should be more systematically considered as an integral part of the measurement of infant looking. We discuss the utility of our scheme to better understand the dynamics of looking and task performance in infant looking paradigms: those involving eye-tracking and those measuring looking duration.

Verbal Interactional Synchronization between Therapist and Children with Autism Spectrum Disorder during Dolphin Assisted Therapy: Five Case Studies

Synchronizing behaviors in interactions, such as during turn-taking, are often impaired in children with Autism Spectrum Disorder. Therapies that focus on turn-taking generally lead to increased social skills, less interruptions, and silent pauses, however a positive non-demanding environment is therefore thought to be beneficial. Such an environment can be achieved by incorporating animals into therapy. Our study was guided by the following research questions: (1) How can we characterize the interaction between child and therapist during dolphin-assisted therapy, with regard to synchrony in verbalizations (turn-taking) and (2) does synchrony change over the course of six sessions of therapy? To answer these questions, we performed a cross-recurrence quantification analysis on behavioral data of five children, to give a detailed view of the interaction between therapist and child in the context of dolphin-assisted therapy. We were able to detect synchrony (i.e., adequate turn-taking) in all dyads, although not all children improved equally. The differences might be explained by a delayed reaction time of some children, and their level of language development.

Beyond fixation durations: Recurrence quantification analysis reveals spatiotemporal dynamics of infant visual scanning

Standard looking-duration measures in eye-tracking data provide only general quantitative indices, while details of the spatiotemporal structuring of fixation sequences are lost. To overcome this, various tools have been developed to measure the dynamics of fixations. However, these analyses are only useful when stimuli have high perceptual similarity and they require the previous definition of areas of interest (AOIs). Although these methods have been widely applied in adult studies, relatively little is known about the temporal structuring of infant gaze-foraging behaviors such as variability of scanning over time or individual scanning patterns. Thus, to shed more light on the spatiotemporal characteristics of infant fixation sequences we apply for the first time a new methodology for nonlinear time-series analysis-the recurrence quantification analysis (RQA). We present how the dynamics of infant scanning varies depending on the scene content during a "pop-out" search task. Moreover, we show how the normalization of RQA measures with average fixation durations provides a more detailed account of the dynamics of fixation sequences. Finally, we link the RQA measures of temporal dynamics of scanning with the spatial information about the stimuli using heat maps of recurrences without the need for defining a priori AOIs and present how infants' foraging strategies are driven by the image content. We conclude from our findings that the RQA methodology has potential applications in the analysis of the temporal dynamics of infant visual foraging offering advantages over existing methods.