The Virtual Teacher (VT) Paradigm: Learning New Patterns of Interpersonal Coordination Using the Human Dynamic Clamp

Bringing together multimodal and multilevel approaches to study the emergence of social bonds between children and improve social AI

This protocol paper outlines an innovative multimodal and multilevel approach to studying the emergence and evolution of how children build social bonds with their peers, and its potential application to improving social artificial intelligence (AI). We detail a unique hyperscanning experimental framework utilizing functional near-infrared spectroscopy (fNIRS) to observe inter-brain synchrony in child dyads during collaborative tasks and social interactions. Our proposed longitudinal study spans middle childhood, aiming to capture the dynamic development of social connections and cognitive engagement in naturalistic settings. To do so we bring together four kinds of data: the multimodal conversational behaviors that dyads of children engage in, evidence of their state of interpersonal rapport, collaborative performance on educational tasks, and inter-brain synchrony. Preliminary pilot data provide foundational support for our approach, indicating promising directions for identifying neural patterns associated with productive social interactions. The planned research will explore the neural correlates of social bond formation, informing the creation of a virtual peer learning partner in the field of Social Neuroergonomics. This protocol promises significant contributions to understanding the neural basis of social connectivity in children, while also offering a blueprint for designing empathetic and effective social AI tools, particularly for educational contexts.

(Mis)align: a simple dynamic framework for modeling interpersonal coordination

As people coordinate in daily interactions, they engage in different patterns of behavior to achieve successful outcomes. This includes both synchrony-the temporal coordination of the same behaviors at the same time-and complementarity-the coordination of the same or different behaviors that may occur at different relative times. Using computational methods, we develop a simple framework to describe the interpersonal dynamics of behavioral synchrony and complementarity over time, and explore their task-dependence. A key feature of this framework is the inclusion of a task context that mediates interactions, and consists of active, inactive, and inhibitory constraints on communication. Initial simulation results show that these task constraints can be a robust predictor of simulated agents' behaviors over time. We also show that the framework can reproduce some general patterns observed in human interaction data. We describe preliminary theoretical implications from these results, and relate them to broader proposals of synergistic self-organization in communication.

Using a haptic dynamic clamp to reduce arousal: preference, arousal, and coordination stability are related

We have developed a haptic dynamic clamp dedicated to the regulation of arousal. It takes the form of a vibrating stress ball to be squeezed, called Viball, controlled by Righetti's nonlinear adaptive Hopf oscillator. Participants squeezed an adaptive Viball which adapts its frequency of vibration to the current frequency of human squeezing. The adaptive Viball was compared to three non-adaptive Viballs, parametrized to vibrate at a lower, equal, or higher frequency than the participants' preferred frequency. While squeezing the ball, participants looked at stressful or calming pictures and their electrodermal activity was recorded. Using the preference paradigm, we show that participants preferred to interact with the adaptive Viball rather than with the most slowly vibrating ball that most strongly reduced arousal. The stability of the human-ball coordination was the highest with the adaptive Viball. There was also a positive correlation between the stability of coordination and arousal. The data are discussed in light of the energy-based interpretation of coordination dynamics.

The spontaneous emergence of rhythmic coordination in turn taking

Turn-taking is a feature of many social interactions such as group music-making, where partners must alternate turns with high precision and accuracy. In two studies of musical rhythm coordination, we investigated how joint action partners learn to coordinate the timing of turn-taking. Musically inexperienced individuals learned to tap at the rate of a pacing cue individually or jointly (in turn with a partner), where each tap produced the next tone in a melodic sequence. In Study 1, partners alternated turns every tap, whereas in Study 2 partners alternated turns every two taps. Findings revealed that partners did not achieve the same level of performance accuracy or precision of inter-tap intervals (ITIs) when producing tapping sequences jointly relative to individually, despite showing learning (increased ITI accuracy and precision across the experiment) in both tasks. Strikingly, partners imposed rhythmic patterns onto jointly produced sequences that captured the temporal structure of turns. Together, learning to produce novel temporal sequences in turn with a partner appears to be more challenging than learning to produce the same sequences alone. Critically, partners may impose rhythmic structures onto turn-taking sequences as a strategy for facilitating coordination.

Reciprocity and alignment: quantifying coupling in dynamic interactions

Recent accounts of social cognition focus on how we do things together, suggesting that becoming aligned relies on a reciprocal exchange of information. The next step is to develop richer computational methods that quantify the degree of coupling and describe the nature of the information exchange. We put forward a definition of coupling, comparing it to related terminology and detail, available computational methods and the level of organization to which they pertain, presenting them as a hierarchy from weakest to richest forms of coupling. The rationale is that a temporally coherent link between two dynamical systems at the lowest level of organization sustains mutual adaptation and alignment at the highest level. Postulating that when we do things together, we do so dynamically over time and we argue that to determine and measure instances of true reciprocity in social exchanges is key. Along with this computationally rich definition of coupling, we present challenges for the field to be tackled by a diverse community working towards a dynamic account of social cognition.

The effect of inherent and incidental constraints on bimanual and social coordination

The current investigation was designed to examine the influence of inherent and incidental constraints on the stability characteristics associated with bimanual and social coordination. Individual participants (N = 9) and pairs of participants (N = 18, 9 pairs) were required to rhythmically coordinate patterns of isometric forces in 1:1 in-phase and 1:2 multi-frequency patterns by exerting force with their right and left limbs. Lissajous information was provided to guide performance. Participants performed 13 practice trials and 1 test trial per pattern. On the test trial, muscle activity from the triceps brachii muscles of each arm was recorded. EMG-EMG coherence between the two EMG signals was calculated using wavelet coherence. The behavioral data indicated that individual participants performed the 1:1 in-phase pattern more accurately and with less variability than paired participants. The EMG coherence analysis indicated significantly higher coherence for individual participants than for the paired participants during the 1:1 in-phase pattern, whereas no differences were observed between groups for the 1:2 coordination pattern. The results of the current investigation support the notion that neural crosstalk can stabilize 1:1 in-phase coordination when contralateral and ipsilateral signals are integrated via the neuromuscular linkage between two effectors.

Coordination Dynamics: A Foundation for Understanding Social Behavior

Humans' interactions with each other or with socially competent machines exhibit lawful coordination patterns at multiple levels of description. According to Coordination Dynamics, such laws specify the flow of coordination states produced by functional synergies of elements (e.g., cells, body parts, brain areas, people…) that are temporarily organized as single, coherent units. These coordinative structures or synergies may be mathematically characterized as informationally coupled self-organizing dynamical systems (Coordination Dynamics). In this paper, we start from a simple foundation, an elemental model system for social interactions, whose behavior has been captured in the Haken-Kelso-Bunz (HKB) model. We follow a tried and tested scientific method that tightly interweaves experimental neurobehavioral studies and mathematical models. We use this method to further develop a body of empirical research that advances the theory toward more generalized forms. In concordance with this interdisciplinary spirit, the present paper is written both as an overview of relevant advances and as an introduction to its mathematical underpinnings. We demonstrate HKB's evolution in the context of social coordination along several directions, with its applicability growing to increasingly complex scenarios. In particular, we show that accommodating for symmetry breaking in intrinsic dynamics and coupling, multiscale generalization and adaptation are principal evolutions. We conclude that a general framework for social coordination dynamics is on the horizon, in which models support experiments with hypothesis generation and mechanistic insights.

The Human Dynamic Clamp Reveals the Fronto-Parietal Network Linking Real-Time Social Coordination and Cognition

How does the brain allow us to interact with others? Social neuroscience has already provided some answers to these questions but has tended to treat high-level, cognitive interpretations of social behavior separately from the sensorimotor mechanisms upon which they rely. The goal here is to identify the underlying neural processes and mechanisms linking sensorimotor coordination and intention attribution. We combine the human dynamic clamp, a novel paradigm for studyingrealistic social behavior, with high-resolution electroencephalography. The collection of humanness and intention attribution reports, kinematics, and neural data affords an opportunity to relate brain activity to the ongoing social behavior. Behavioral results demonstrate that sensorimotor coordination influences the judgments of cooperativeness and humanness. Analysis of brain dynamics reveals two distinct networks related to the integration of visuo-motor information from self and other which overlap over the right parietal region. Furthermore, judgment of humanness and cooperation of others modulate the functional connectivity between this right parietal hub and the prefrontal cortex. These results reveal how distributed neural dynamics integrates information from "low-level" sensorimotor mechanisms and "high-level" social cognition to support the realistic social behaviors that play out in real time during interactive scenarios.

Interactive Psychometrics for Autism With the Human Dynamic Clamp: Interpersonal Synchrony From Sensorimotor to Sociocognitive Domains

The human dynamic clamp (HDC) is a human-machine interface designed on the basis of coordination dynamics for studying realistic social interaction under controlled and reproducible conditions. Here, we propose to probe the validity of the HDC as a psychometric instrument for quantifying social abilities in children with autism spectrum disorder (ASD) and neurotypical development. To study interpersonal synchrony with the HDC, we derived five standardized scores following a gradient from sensorimotor and motor to higher sociocognitive skills in a sample of 155 individuals (113 participants with ASD, 42 typically developing participants; aged 5 to 25 years; IQ > 70). Regression analyses were performed using normative modeling on global scores according to four subconditions (HDC behavior "cooperative/competitive," human task "in-phase/anti-phase," diagnosis, and age at inclusion). Children with ASD had lower scores than controls for motor skills. HDC motor coordination scores were the best candidates for stratification and diagnostic biomarkers according to exploratory analyses of hierarchical clustering and multivariate classification. Independently of phenotype, sociocognitive skills increased with developmental age while being affected by the ongoing task and HDC behavior. Weaker performance in ASD for motor skills suggests the convergent validity of the HDC for evaluating social interaction. Results provided additional evidence of a relationship between sensorimotor and sociocognitive skills. HDC may also be used as a marker of maturation of sociocognitive skills during real-time social interaction. Through its standardized and objective evaluation, the HDC not only represents a valid paradigm for the study of interpersonal synchrony but also offers a promising, clinically relevant psychometric instrument for the evaluation and stratification of sociomotor dysfunctions.

Feedback delays can enhance anticipatory synchronization in human-machine interaction

Research investigating the dynamics of coupled physical systems has demonstrated that small feedback delays can allow a dynamic response system to anticipate chaotic behavior. This counterintuitive phenomenon, termed anticipatory synchronization, has been observed in coupled electrical circuits, laser semi-conductors, and artificial neurons. Recent research indicates that the same process might also support the ability of humans to anticipate the occurrence of chaotic behavior in other individuals. Motivated by this latter work, the current study examined whether the process of feedback delay induced anticipatory synchronization could be employed to develop an interactive artificial agent capable of anticipating chaotic human movement. Results revealed that incorporating such delays within the movement-control dynamics of an artificial agent not only enhances an artificial agent’s ability to anticipate chaotic human behavior, but to synchronize with such behavior in a manner similar to natural human-human anticipatory synchronization. The implication of these findings for the development of human-machine interaction systems is discussed.

Teacher-learner interaction quantifies scaffolding behaviour in imitation learning

Teachers often believe that they take into account learners’ ongoing learning progress in their teaching. Can behavioural data support this belief? To address this question, we investigated the interactive behavioural coordination between teachers and learners during imitation learning to solve a puzzle. The teacher manually demonstrated the puzzle solution to a learner who immediately imitated and learned it. Manual movements of teachers and learners were analysed using a bivariate autoregressive model. To identify bidirectional information exchange and information shared between the two agents, we calculated causality and noise covariance from the model. Information transfer observed from teacher to learner in the lateral component of their motion indicated imitation of the spatial information of the puzzle solution. Information transfer from learner to teacher in the vertical component of their motion indicated the monitoring process through which teachers adjust their timing of demonstration to the learner’s progress. The shared information in the lateral component increased as learning progressed, indicating the knowledge was shared between the two agents. Our findings demonstrated that the teacher interactively engaged in and contingently supported (i.e. scaffolded) imitation. We thus provide a behavioural signature of the teacher’s intention to promote learning indispensable for understanding the nature of teaching.

Human social motor solutions for human-machine interaction in dynamical task contexts

Multiagent activity is commonplace in everyday life and can improve the behavioral efficiency of task performance and learning. Thus, augmenting social contexts with the use of interactive virtual and robotic agents is of great interest across health, sport, and industry domains. However, the effectiveness of human-machine interaction (HMI) to effectively train humans for future social encounters depends on the ability of artificial agents to respond to human coactors in a natural, human-like manner. One way to achieve effective HMI is by developing dynamical models utilizing dynamical motor primitives (DMPs) of human multiagent coordination that not only capture the behavioral dynamics of successful human performance but also, provide a tractable control architecture for computerized agents. Previous research has demonstrated how DMPs can successfully capture human-like dynamics of simple nonsocial, single-actor movements. However, it is unclear whether DMPs can be used to model more complex multiagent task scenarios. This study tested this human-centered approach to HMI using a complex dyadic shepherding task, in which pairs of coacting agents had to work together to corral and contain small herds of virtual sheep. Human-human and human-artificial agent dyads were tested across two different task contexts. The results revealed (i) that the performance of human-human dyads was equivalent to those composed of a human and the artificial agent and (ii) that, using a "Turing-like" methodology, most participants in the HMI condition were unaware that they were working alongside an artificial agent, further validating the isomorphism of human and artificial agent behavior.

Joint dyadic action: Error correction by two persons works better than by one alone

We investigated how two people learn to coordinate their movement to achieve a joint goal. Pairs of participants oscillated a joystick with their dominant hand whilst looking at a common feedback, a Lissajous figure, where each participant controlled either the vertical or horizontal coordinate of a moving dot. In the absence of specific instructions, inter-personal coordination was highly variable, punctuated by intermittent phase locking. When participants were required to produce a circular Lissajous figure, coordination variability decreased while accuracy, transfer entropy and the incidence of stable coordinative solutions (fixed points, including bi-stability) increased as a function of practice trials. When one partner closed his/her eyes, so that the other one received the full control of error correction, the stability and accuracy of coordination decreased. A questionnaire showed that partners experienced the feeling of we-control. The results were interpreted in terms of a disturbance ∼ correction challenge: joint action is enhanced by having a flexibly adjusting co-actor rather than a more predictable, but not adjusting, partner. At transfer, partners were able to produce a new, never-practiced Lissajous pattern, evidencing the generalisability of joint learning.

Critical diversity: Divided or united states of social coordination

Much of our knowledge of coordination comes from studies of simple, dyadic systems or systems containing large numbers of components. The huge gap 'in between' is seldom addressed, empirically or theoretically. We introduce a new paradigm to study the coordination dynamics of such intermediate-sized ensembles with the goal of identifying key mechanisms of interaction. Rhythmic coordination was studied in ensembles of eight people, with differences in movement frequency ('diversity') manipulated within the ensemble. Quantitative change in diversity led to qualitative changes in coordination, a critical value separating régimes of integration and segregation between groups. Metastable and multifrequency coordination between participants enabled communication across segregated groups within the ensemble, without destroying overall order. These novel findings reveal key factors underlying coordination in ensemble sizes previously considered too complicated or 'messy' for systematic study and supply future theoretical/computational models with new empirical checkpoints.

How Interpersonal Coordination Affects Individual Behavior (and Vice Versa): Experimental analysis and adaptive HKB model of social memory

How one behaves after interacting with a friend may not be the same as before the interaction. The present study investigated which spontaneous coordination patterns formed between two persons and whether a remnant of the interaction remained ("social memory"). Pairs of people sat face-to-face and continuously flexed index fingers while vision between partners was manipulated to allow or prevent information exchange. Trials consisted of three successive twenty-second intervals: without vision, with vision, and again without vision. Steady, transient, or absent phase coupling was observed during vision. In support of social memory, participants tended to remain near each other's movement frequency after the interaction ended. Furthermore, the greater the stability of interpersonal coordination, the more similar partners' post-interactional frequencies became. Proposing that social memory resulted from prior frequency adaptation, a model based on Haken-Kelso-Bunz oscillators reproduced the experimental findings, even for patterns observed on individual trials. Parametric manipulations revealed multiple routes to social memory through the interplay of adaptation and other model parameters. The experimental results, model, and interpretation motivate potential future research and therapeutic applications.

A roadmap to computational social neuroscience

To complement experimental efforts toward understanding human social interactions at both neural and behavioral levels, two computational approaches are presented: (1) a fully parameterizable mathematical model of a social partner, the Human Dynamic Clamp which, by virtue of experimentally controlled interactions between Virtual Partners and real people, allows for emergent behaviors to be studied; and (2) a multiscale neurocomputational model of social coordination that enables exploration of social self-organization at all levels-from neuronal patterns to people interacting with each other. These complementary frameworks and the cross product of their analysis aim at understanding the fundamental principles governing social behavior.

The co-creation of meaningful action: bridging enaction and interactional sociology

What makes possible the co-creation of meaningful action? In this paper, we go in search of an answer to this question by combining insights from interactional sociology and enaction. Both research schools investigate social interactions as such, and conceptualize their organization in terms of autonomy. We ask what it could mean for an interaction to be autonomous, and discuss the structures and processes that contribute to and are maintained in the so-called interaction order. We also discuss the role played by individual vulnerability as well as the vulnerability of social interaction processes in the co-creation of meaningful action. Finally, we outline some implications of this interdisciplinary fraternization for the empirical study of social understanding, in particular in social neuroscience and psychology, pointing out the need for studies based on dynamic systems approaches on origins and references of coordination, and experimental designs to help understand human co-presence.

Design of a Virtual Player for Joint Improvisation with Humans in the Mirror Game

Joint improvisation is often observed among humans performing joint action tasks. Exploring the underlying cognitive and neural mechanisms behind the emergence of joint improvisation is an open research challenge. This paper investigates jointly improvised movements between two participants in the mirror game, a paradigmatic joint task example. First, experiments involving movement coordination of different dyads of human players are performed in order to build a human benchmark. No designation of leader and follower is given beforehand. We find that joint improvisation is characterized by the lack of a leader and high levels of movement synchronization. Then, a theoretical model is proposed to capture some features of their interaction, and a set of experiments is carried out to test and validate the model ability to reproduce the experimental observations. Furthermore, the model is used to drive a computer avatar able to successfully improvise joint motion with a human participant in real time. Finally, a convergence analysis of the proposed model is carried out to confirm its ability to reproduce joint movements between the participants.

Enhanced emotional responses during social coordination with a virtual partner

Emotion and motion, though seldom studied in tandem, are complementary aspects of social experience. This study investigates variations in emotional responses during movement coordination between a human and a Virtual Partner (VP), an agent whose virtual finger movements are driven by the Haken-Kelso-Bunz (HKB) equations of Coordination Dynamics. Twenty-one subjects were instructed to coordinate finger movements with the VP in either inphase or antiphase patterns. By adjusting model parameters, we manipulated the 'intention' of VP as cooperative or competitive with the human's instructed goal. Skin potential responses (SPR) were recorded to quantify the intensity of emotional response. At the end of each trial, subjects rated the VP's intention and whether they thought their partner was another human being or a machine. We found greater emotional responses when subjects reported that their partner was human and when coordination was stable. That emotional responses are strongly influenced by dynamic features of the VP's behavior, has implications for mental health, brain disorders and the design of socially cooperative machines.