Virtual Partner Interaction (VPI): exploring novel behaviors via coordination dynamics

Spontaneity competes with intention to influence the coordination dynamics of interpersonal performance tendencies

Research has shown that spontaneous visual coupling supports frequency entrainment, phase attraction, and intermittent interpersonal coordination when co-actors are switched from a no-vision (NV) to vision (V) context. In two experiments, co-actors started in a NV context while producing the same or different amplitude movements. The same amplitude resulted in similar self-paced frequencies, while different amplitudes resulted in disparate frequencies. In experiment 1, co-actors were instructed to maintain amplitude while receiving no instructions to coordinate their actions. Frequency and phase entrainment was limited in the V context even when co-actors started the NV context with the same amplitude. In experiment 2, co-actors were instructed to maintain amplitude and intentionally coordinate together, but not at a specific pattern. Significant frequency modulations occurred to maintain amplitude as the co-actors sought to coordinate their actions. With the open-ended instructions, co-actors produced in-phase and anti-phase coordination along with intermittent performance exhibited by shifts between a variety of stable relative phase patterns. The proposed hypotheses and findings are discussed within the context of a shared manifold representation for joint action contexts, with the coordination dynamics expressed by the HKB model of relative phase serving to conceptualization the representations in the shared manifold.

Exploring the dynamics of intentional sensorimotor desynchronization using phasing performance in music

Humans tend to synchronize spontaneously to rhythmic stimuli or with other humans, but they can also desynchronize intentionally in certain situations. In this study, we investigate the dynamics of intentional sensorimotor desynchronization using phasing performance in music as an experimental paradigm. Phasing is a compositional technique in modern music that requires musicians to desynchronize from each other in a controlled manner. A previous case study found systematic nonlinear trajectories in the phasing performance between two expert musicians, which were explained by coordination dynamics arising from the interaction between the intrinsic tendency of synchronization and the intention of desynchronization. A recent exploratory study further examined the dynamics of phasing performance using a simplified task of phasing against a metronome. Here we present a further analysis and modeling of the data from the exploratory study, focusing on the various types of phasing behavior found in non-expert participants. Participants were instructed to perform one phasing lap, and individual trials were classified as successful (1 lap), unsuccessful (> 1 laps), or incomplete (0 lap) based on the number of laps made. It was found that successful phasing required a gradual increment of relative phase and that different types of failure (unsuccessful vs. incomplete) were prevalent at slow vs. fast metronome tempi. The results are explained from a dynamical systems perspective, and a dynamical model of phasing performance is proposed which captures the interaction of intrinsic dynamics and intentional control in an adaptive-frequency oscillator coupled to a periodic external stimulus. It is shown that the model can replicate the multiple types of phasing behavior as well as the effect of tempo observed in the human experiment. This study provides further evidence that phasing performance is governed by the nonlinear dynamics of rhythmic coordination. It also demonstrates that the musical technique of phasing provides a unique experimental paradigm for investigating human rhythmic behavior.

Using a haptic dynamic clamp to reduce arousal: preference, arousal, and coordination stability are related

We have developed a haptic dynamic clamp dedicated to the regulation of arousal. It takes the form of a vibrating stress ball to be squeezed, called Viball, controlled by Righetti's nonlinear adaptive Hopf oscillator. Participants squeezed an adaptive Viball which adapts its frequency of vibration to the current frequency of human squeezing. The adaptive Viball was compared to three non-adaptive Viballs, parametrized to vibrate at a lower, equal, or higher frequency than the participants' preferred frequency. While squeezing the ball, participants looked at stressful or calming pictures and their electrodermal activity was recorded. Using the preference paradigm, we show that participants preferred to interact with the adaptive Viball rather than with the most slowly vibrating ball that most strongly reduced arousal. The stability of the human-ball coordination was the highest with the adaptive Viball. There was also a positive correlation between the stability of coordination and arousal. The data are discussed in light of the energy-based interpretation of coordination dynamics.

Artificial Partners to Understand Joint Action: Representing Others to Develop Effective Coordination

In the last years, artificial partners have been proposed as tools to study joint action, as they would allow to address joint behaviors in more controlled experimental conditions. Here we present an artificial partner architecture which is capable of integrating all the available information about its human counterpart and to develop efficient and natural forms of coordination. The model uses an extended state observer which combines prior information, motor commands and sensory observations to infer the partner's ongoing actions (partner model). Over trials, these estimates are gradually incorporated into action selection. Using a joint planar task in which the partners are required to perform reaching movements while mechanically coupled, we demonstrate that the artificial partner develops an internal representation of its human counterpart, whose accuracy depends on the degree of mechanical coupling and on the reliability of the sensory information. We also show that human-artificial dyads develop coordination strategies which closely resemble those observed in human-human dyads and can be interpreted as Nash equilibria. The proposed approach may provide insights for the understanding of the mechanisms underlying human-human interaction. Further, it may inform the development of novel neuro-rehabilitative solutions and more efficient human-machine interfaces.

Integration and Causality in Enactive Approaches to Psychiatry

In this paper I address what has been called the integration problem in psychiatry. This problem is tied to conceptions of causality and explanatory levels in our understanding of mind. I take an interdisciplinary enactive perspective to develop a 3-fold method for exploring the dynamics of integration, based on a concept of dynamical causation and a non-hierarchical (level-free) notion of gestalt. I also consider Autism Spectrum Disorder (ASD) as a test case.

Two dimensionless parameters and a mechanical analogue for the HKB model of motor coordination

The widely cited Haken-Kelso-Bunz (HKB) model of motor coordination is used in an enormous range of applications. In this paper, we show analytically that the weakly damped, weakly coupled HKB model of two oscillators depends on only two dimensionless parameters; the ratio of the linear damping coefficient and the linear coupling coefficient and the ratio of the combined nonlinear damping coefficients and the combined nonlinear coupling coefficients. We illustrate our results with a mechanical analogue. We use our analytic results to predict behaviours in arbitrary parameter regimes and show how this led us to explain and extend recent numerical continuation results of the full HKB model. The key finding is that the HKB model contains a significant amount of behaviour in biologically relevant parameter regimes not yet observed in experiments or numerical simulations. This observation has implications for the development of virtual partner interaction and the human dynamic clamp, and potentially for the HKB model itself.

The Haken-Kelso-Bunz (HKB) model: from matter to movement to mind

This article presents a brief retrospective on the Haken-Kelso-Bunz (HKB) model of certain dynamical properties of human movement. Though unanticipated, HKB introduced, and demonstrated the power of, a new vocabulary for understanding behavior, cognition and the brain, revealed through a visually compelling mathematical picture that accommodated highly reproducible experimental facts and predicted new ones. HKB stands as a harbinger of paradigm change in several scientific fields, the effects of which are still being felt. In particular, HKB constitutes the foundation of a mechanistic science of coordination called Coordination Dynamics that extends from matter to movement to mind, and beyond.

Reciprocity and alignment: quantifying coupling in dynamic interactions

Recent accounts of social cognition focus on how we do things together, suggesting that becoming aligned relies on a reciprocal exchange of information. The next step is to develop richer computational methods that quantify the degree of coupling and describe the nature of the information exchange. We put forward a definition of coupling, comparing it to related terminology and detail, available computational methods and the level of organization to which they pertain, presenting them as a hierarchy from weakest to richest forms of coupling. The rationale is that a temporally coherent link between two dynamical systems at the lowest level of organization sustains mutual adaptation and alignment at the highest level. Postulating that when we do things together, we do so dynamically over time and we argue that to determine and measure instances of true reciprocity in social exchanges is key. Along with this computationally rich definition of coupling, we present challenges for the field to be tackled by a diverse community working towards a dynamic account of social cognition.

Brain Entropy During Aging Through a Free Energy Principle Approach

Neural complexity and brain entropy (BEN) have gained greater interest in recent years. The dynamics of neural signals and their relations with information processing continue to be investigated through different measures in a variety of noteworthy studies. The BEN of spontaneous neural activity decreases during states of reduced consciousness. This evidence has been showed in primary consciousness states, such as psychedelic states, under the name of "the entropic brain hypothesis." In this manuscript we propose an extension of this hypothesis to physiological and pathological aging. We review this particular facet of the complexity of the brain, mentioning studies that have investigated BEN in primary consciousness states, and extending this view to the field of neuroaging with a focus on resting-state functional Magnetic Resonance Imaging. We first introduce historic and conceptual ideas about entropy and neural complexity, treating the mindbrain as a complex nonlinear dynamic adaptive system, in light of the free energy principle. Then, we review the studies in this field, analyzing the idea that the aim of the neurocognitive system is to maintain a dynamic state of balance between order and chaos, both in terms of dynamics of neural signals and functional connectivity. In our exploration we will review studies both on acute psychedelic states and more chronic psychotic states and traits, such as those in schizophrenia, in order to show the increase of entropy in those states. Then we extend our exploration to physiological and pathological aging, where BEN is reduced. Finally, we propose an interpretation of these results, defining a general trend of BEN in primary states and cognitive aging.

Coordination Dynamics: A Foundation for Understanding Social Behavior

Humans' interactions with each other or with socially competent machines exhibit lawful coordination patterns at multiple levels of description. According to Coordination Dynamics, such laws specify the flow of coordination states produced by functional synergies of elements (e.g., cells, body parts, brain areas, people…) that are temporarily organized as single, coherent units. These coordinative structures or synergies may be mathematically characterized as informationally coupled self-organizing dynamical systems (Coordination Dynamics). In this paper, we start from a simple foundation, an elemental model system for social interactions, whose behavior has been captured in the Haken-Kelso-Bunz (HKB) model. We follow a tried and tested scientific method that tightly interweaves experimental neurobehavioral studies and mathematical models. We use this method to further develop a body of empirical research that advances the theory toward more generalized forms. In concordance with this interdisciplinary spirit, the present paper is written both as an overview of relevant advances and as an introduction to its mathematical underpinnings. We demonstrate HKB's evolution in the context of social coordination along several directions, with its applicability growing to increasingly complex scenarios. In particular, we show that accommodating for symmetry breaking in intrinsic dynamics and coupling, multiscale generalization and adaptation are principal evolutions. We conclude that a general framework for social coordination dynamics is on the horizon, in which models support experiments with hypothesis generation and mechanistic insights.

The Human Dynamic Clamp Reveals the Fronto-Parietal Network Linking Real-Time Social Coordination and Cognition

How does the brain allow us to interact with others? Social neuroscience has already provided some answers to these questions but has tended to treat high-level, cognitive interpretations of social behavior separately from the sensorimotor mechanisms upon which they rely. The goal here is to identify the underlying neural processes and mechanisms linking sensorimotor coordination and intention attribution. We combine the human dynamic clamp, a novel paradigm for studyingrealistic social behavior, with high-resolution electroencephalography. The collection of humanness and intention attribution reports, kinematics, and neural data affords an opportunity to relate brain activity to the ongoing social behavior. Behavioral results demonstrate that sensorimotor coordination influences the judgments of cooperativeness and humanness. Analysis of brain dynamics reveals two distinct networks related to the integration of visuo-motor information from self and other which overlap over the right parietal region. Furthermore, judgment of humanness and cooperation of others modulate the functional connectivity between this right parietal hub and the prefrontal cortex. These results reveal how distributed neural dynamics integrates information from "low-level" sensorimotor mechanisms and "high-level" social cognition to support the realistic social behaviors that play out in real time during interactive scenarios.

Speech Sound Disorders in Children: An Articulatory Phonology Perspective

Speech Sound Disorders (SSDs) is a generic term used to describe a range of difficulties producing speech sounds in children (McLeod and Baker, 2017). The foundations of clinical assessment, classification and intervention for children with SSD have been heavily influenced by psycholinguistic theory and procedures, which largely posit a firm boundary between phonological processes and phonetics/articulation (Shriberg, 2010). Thus, in many current SSD classification systems the complex relationships between the etiology (distal), processing deficits (proximal) and the behavioral levels (speech symptoms) is under-specified (Terband et al., 2019a). It is critical to understand the complex interactions between these levels as they have implications for differential diagnosis and treatment planning (Terband et al., 2019a). There have been some theoretical attempts made towards understanding these interactions (e.g., McAllister Byun and Tessier, 2016) and characterizing speech patterns in children either solely as the product of speech motor performance limitations or purely as a consequence of phonological/grammatical competence has been challenged (Inkelas and Rose, 2007; McAllister Byun, 2012). In the present paper, we intend to reconcile the phonetic-phonology dichotomy and discuss the interconnectedness between these levels and the nature of SSDs using an alternative perspective based on the notion of an articulatory "gesture" within the broader concepts of the Articulatory Phonology model (AP; Browman and Goldstein, 1992). The articulatory "gesture" serves as a unit of phonological contrast and characterization of the resulting articulatory movements (Browman and Goldstein, 1992; van Lieshout and Goldstein, 2008). We present evidence supporting the notion of articulatory gestures at the level of speech production and as reflected in control processes in the brain and discuss how an articulatory "gesture"-based approach can account for articulatory behaviors in typical and disordered speech production (van Lieshout, 2004; Pouplier and van Lieshout, 2016). Specifically, we discuss how the AP model can provide an explanatory framework for understanding SSDs in children. Although other theories may be able to provide alternate explanations for some of the issues we will discuss, the AP framework in our view generates a unique scope that covers linguistic (phonology) and motor processes in a unified manner.

Interactive Psychometrics for Autism With the Human Dynamic Clamp: Interpersonal Synchrony From Sensorimotor to Sociocognitive Domains

The human dynamic clamp (HDC) is a human-machine interface designed on the basis of coordination dynamics for studying realistic social interaction under controlled and reproducible conditions. Here, we propose to probe the validity of the HDC as a psychometric instrument for quantifying social abilities in children with autism spectrum disorder (ASD) and neurotypical development. To study interpersonal synchrony with the HDC, we derived five standardized scores following a gradient from sensorimotor and motor to higher sociocognitive skills in a sample of 155 individuals (113 participants with ASD, 42 typically developing participants; aged 5 to 25 years; IQ > 70). Regression analyses were performed using normative modeling on global scores according to four subconditions (HDC behavior "cooperative/competitive," human task "in-phase/anti-phase," diagnosis, and age at inclusion). Children with ASD had lower scores than controls for motor skills. HDC motor coordination scores were the best candidates for stratification and diagnostic biomarkers according to exploratory analyses of hierarchical clustering and multivariate classification. Independently of phenotype, sociocognitive skills increased with developmental age while being affected by the ongoing task and HDC behavior. Weaker performance in ASD for motor skills suggests the convergent validity of the HDC for evaluating social interaction. Results provided additional evidence of a relationship between sensorimotor and sociocognitive skills. HDC may also be used as a marker of maturation of sociocognitive skills during real-time social interaction. Through its standardized and objective evaluation, the HDC not only represents a valid paradigm for the study of interpersonal synchrony but also offers a promising, clinically relevant psychometric instrument for the evaluation and stratification of sociomotor dysfunctions.

Delayed feedback embedded in perception-action coordination cycles results in anticipation behavior during synchronized rhythmic action: A dynamical systems approach

Dancing and playing music require people to coordinate actions with auditory rhythms. In laboratory perception-action coordination tasks, people are asked to synchronize taps with a metronome. When synchronizing with a metronome, people tend to anticipate stimulus onsets, tapping slightly before the stimulus. The anticipation tendency increases with longer stimulus periods of up to 3500ms, but is less pronounced in trained individuals like musicians compared to non-musicians. Furthermore, external factors influence the timing of tapping. These factors include the presence of auditory feedback from one’s own taps, the presence of a partner performing coordinated joint tapping, and transmission latencies (TLs) between coordinating partners. Phenomena like the anticipation tendency can be explained by delay-coupled systems, which may be inherent to the sensorimotor system during perception-action coordination. Here we tested whether a dynamical systems model based on this hypothesis reproduces observed patterns of human synchronization. We simulated behavior with a model consisting of an oscillator receiving its own delayed activity as input. Three simulation experiments were conducted using previously-published behavioral data from 1) simple tapping, 2) two-person alternating beat-tapping, and 3) two-person alternating rhythm-clapping in the presence of a range of constant auditory TLs. In Experiment 1, our model replicated the larger anticipation observed for longer stimulus intervals and adjusting the amplitude of the delayed feedback reproduced the difference between musicians and non-musicians. In Experiment 2, by connecting two models we replicated the smaller anticipation observed in human joint tapping with bi-directional auditory feedback compared to joint tapping without feedback. In Experiment 3, we varied TLs between two models alternately receiving signals from one another. Results showed reciprocal lags at points of alternation, consistent with behavioral patterns. Overall, our model explains various anticipatory behaviors, and has potential to inform theories of adaptive human synchronization.

When navigating a busy sidewalk, people coordinate their behavior in an orderly manner. Other activities require people to carefully synchronize periodic actions, as in a group rowing or marching. When individuals tap in synchrony with a metronome, their taps tend to anticipate the metronome. Experiments have revealed that factors like musical expertise, the presence of a synchronizing partner, auditory feedback, and the sound travel time, all systematically affect the tendency to anticipate. While researchers have hypothesized a number of potential mechanisms for such anticipatory behavior, none have successfully accounted for all of the effects. Previous research on coupled physical systems has shown that when one system receives input from a second system, plus its own delayed signal as input, this causes system 1 to anticipate system 2. We hypothesize that the tendency to anticipate is the result of delayed communication between neurons. Our work demonstrates the ability of delay-coupled physical systems to capture human anticipation and the effect of external factors in the anticipation tendency. Our model supports the theory that delayed communication within the nervous system is crucial to understanding anticipatory coordinative behavior.

Connecting empirical phenomena and theoretical models of biological coordination across scales

Coordination in living systems-from cells to people-must be understood at multiple levels of description. Analyses and modelling of empirically observed patterns of biological coordination often focus either on ensemble-level statistics in large-scale systems with many components, or on detailed dynamics in small-scale systems with few components. The two approaches have proceeded largely independent of each other. To bridge this gap between levels and scales, we have recently conducted a human experiment of mid-scale social coordination specifically designed to reveal coordination at multiple levels (ensemble, subgroups and dyads) simultaneously. Based on this experiment, the present work shows that, surprisingly, a single system of equations captures key observations at all relevant levels. It also connects empirically validated models of large- and small-scale biological coordination-the Kuramoto and extended Haken-Kelso-Bunz (HKB) models-and the hallmark phenomena that each is known to capture. For example, it exhibits both multistability and metastability observed in small-scale empirical research (via the second-order coupling and symmetry breaking in extended HKB) and the growth of biological complexity as a function of scale (via the scalability of the Kuramoto model). Only by incorporating both of these features simultaneously can we reproduce the essential coordination behaviour observed in our experiment.

Feedback delays can enhance anticipatory synchronization in human-machine interaction

Research investigating the dynamics of coupled physical systems has demonstrated that small feedback delays can allow a dynamic response system to anticipate chaotic behavior. This counterintuitive phenomenon, termed anticipatory synchronization, has been observed in coupled electrical circuits, laser semi-conductors, and artificial neurons. Recent research indicates that the same process might also support the ability of humans to anticipate the occurrence of chaotic behavior in other individuals. Motivated by this latter work, the current study examined whether the process of feedback delay induced anticipatory synchronization could be employed to develop an interactive artificial agent capable of anticipating chaotic human movement. Results revealed that incorporating such delays within the movement-control dynamics of an artificial agent not only enhances an artificial agent’s ability to anticipate chaotic human behavior, but to synchronize with such behavior in a manner similar to natural human-human anticipatory synchronization. The implication of these findings for the development of human-machine interaction systems is discussed.

Tutorial and simulations with ADAM: an adaptation and anticipation model of sensorimotor synchronization

Interpersonal coordination of movements often involves precise synchronization of action timing, particularly in expert domains such as ensemble music performance. According to the adaptation and anticipation model (ADAM) of sensorimotor synchronization, precise yet flexible interpersonal coordination is supported by reactive error correction mechanisms and anticipatory mechanisms that exploit systematic patterns in stimulus timing to plan future actions. Here, we provide a tutorial introduction to the computational architecture of ADAM and present a series of single- and dual-virtual agent simulations that examine the model parameters that produce ideal synchronization performance in different tempo conditions. In the single-agent simulations, a virtual agent synchronized responses to steady tempo sequence or a sequence containing gradual tempo changes. Parameters controlling basic reactive error (phase) correction were sufficient for producing ideal synchronization performance at the steady tempo, whereas parameters controlling anticipatory mechanisms were necessary for ideal performance with a tempo-changing sequence. In the dual-agent simulations, two interacting virtual agents produced temporal sequences from either congruent or incongruent internal performance templates specifying a steady tempo or tempo changes. Ideal performance was achieved with reactive error correction alone when both agents implemented the same performance template (either steady tempo or tempo change). In contrast, anticipatory mechanisms played a key role when one agent implemented a steady tempo template and the other agent implemented a tempo change template. These findings have implications for understanding the interplay between reactive and anticipatory mechanisms when agents possess compatible versus incompatible representations of task goals during human-human and human-machine interaction.

Walking with avatars: Gait-related visual information for following a virtual leader

Dynamic situations, such as interactive sports or walking on a busy street, impose high demands on a person's ability to interact with (others in) its environment (i.e., 'interact-ability'). The current study examined how distance regulation, a fundamental component of these interactions, is mediated by different sources of visual information. Participants were presented with a back and forwards moving virtual leader, which they had to follow by walking back and forwards themselves. We presented the leader in several appearances that differed in the presence of segmental (i.e., relative movements of body segments), cadence-related (i.e., sway and bounce), and global (i.e., optical expansion-compression) information. Results indicated that removing segmental motion information from the virtual leader significantly deteriorated both temporal synchronization and spatial accuracy of the follower to the leader, especially when the movement path of the leader was less regular/predictable. However, no difference was found between cadence-related and global motion information appearances. We argue that regulating distance with others effectively requires a versatile attunement to segmental and global motion information depending on the specific task demands. The results further support the notion that detection of especially segmental information allows for more timely 'anticipatory' tuning to another person's locomotor movements and intentions.

Human social motor solutions for human-machine interaction in dynamical task contexts

Multiagent activity is commonplace in everyday life and can improve the behavioral efficiency of task performance and learning. Thus, augmenting social contexts with the use of interactive virtual and robotic agents is of great interest across health, sport, and industry domains. However, the effectiveness of human-machine interaction (HMI) to effectively train humans for future social encounters depends on the ability of artificial agents to respond to human coactors in a natural, human-like manner. One way to achieve effective HMI is by developing dynamical models utilizing dynamical motor primitives (DMPs) of human multiagent coordination that not only capture the behavioral dynamics of successful human performance but also, provide a tractable control architecture for computerized agents. Previous research has demonstrated how DMPs can successfully capture human-like dynamics of simple nonsocial, single-actor movements. However, it is unclear whether DMPs can be used to model more complex multiagent task scenarios. This study tested this human-centered approach to HMI using a complex dyadic shepherding task, in which pairs of coacting agents had to work together to corral and contain small herds of virtual sheep. Human-human and human-artificial agent dyads were tested across two different task contexts. The results revealed (i) that the performance of human-human dyads was equivalent to those composed of a human and the artificial agent and (ii) that, using a "Turing-like" methodology, most participants in the HMI condition were unaware that they were working alongside an artificial agent, further validating the isomorphism of human and artificial agent behavior.

A flexible and accurate method to estimate the mode and stability of spontaneous coordinated behaviors: The index-of-stability (IS) analysis

Patterns of coordination result from the interaction between (at least) two oscillatory components. This interaction is typically understood by means of two variables: the mode that expresses the shape of the interaction, and the stability that is the robustness of the interaction in this mode. A potent method of investigating coordinated behaviors is to examine the extent to which patterns of coordination arise spontaneously. However, a prominent issue faced by researchers is that, to date, no standard methods exist to fairly assess the stability of spontaneous coordination. In the present study, we introduce a new method called the index-of-stability (IS) analysis. We developed this method from the phase-coupling (PC) analysis that has been traditionally used for examining locomotion-respiration coordinated systems. We compared the extents to which both methods estimate the stability of simulated coordinated behaviors. Computer-generated time series were used to simulate the coordination of two rhythmic components according to a selected mode m:n and a selected degree of stability. The IS analysis was superior to the PC analysis in estimating the stability of spontaneous coordinated behaviors, in three ways: First, the estimation of stability itself was found to be more accurate and more reliable with the IS analysis. Second, the IS analysis is not constrained by the limitations of the PC analysis. Third and last, the IS analysis offers more flexibility, and so can be adapted according to the user's needs.

Critical diversity: Divided or united states of social coordination

Much of our knowledge of coordination comes from studies of simple, dyadic systems or systems containing large numbers of components. The huge gap 'in between' is seldom addressed, empirically or theoretically. We introduce a new paradigm to study the coordination dynamics of such intermediate-sized ensembles with the goal of identifying key mechanisms of interaction. Rhythmic coordination was studied in ensembles of eight people, with differences in movement frequency ('diversity') manipulated within the ensemble. Quantitative change in diversity led to qualitative changes in coordination, a critical value separating régimes of integration and segregation between groups. Metastable and multifrequency coordination between participants enabled communication across segregated groups within the ensemble, without destroying overall order. These novel findings reveal key factors underlying coordination in ensemble sizes previously considered too complicated or 'messy' for systematic study and supply future theoretical/computational models with new empirical checkpoints.

How Interpersonal Coordination Affects Individual Behavior (and Vice Versa): Experimental analysis and adaptive HKB model of social memory

How one behaves after interacting with a friend may not be the same as before the interaction. The present study investigated which spontaneous coordination patterns formed between two persons and whether a remnant of the interaction remained ("social memory"). Pairs of people sat face-to-face and continuously flexed index fingers while vision between partners was manipulated to allow or prevent information exchange. Trials consisted of three successive twenty-second intervals: without vision, with vision, and again without vision. Steady, transient, or absent phase coupling was observed during vision. In support of social memory, participants tended to remain near each other's movement frequency after the interaction ended. Furthermore, the greater the stability of interpersonal coordination, the more similar partners' post-interactional frequencies became. Proposing that social memory resulted from prior frequency adaptation, a model based on Haken-Kelso-Bunz oscillators reproduced the experimental findings, even for patterns observed on individual trials. Parametric manipulations revealed multiple routes to social memory through the interplay of adaptation and other model parameters. The experimental results, model, and interpretation motivate potential future research and therapeutic applications.

Design and Validation of a Virtual Player for Studying Interpersonal Coordination in the Mirror Game

The mirror game has been recently proposed as a simple, yet powerful paradigm for studying interpersonal interactions. It has been suggested that a virtual partner able to play the game with human subjects can be an effective tool to affect the underlying neural processes needed to establish the necessary connections between the players, and also to provide new clinical interventions for rehabilitation of patients suffering from social disorders. Inspired by the motor processes of the central nervous system (CNS) and the musculoskeletal system in the human body, in this paper we develop a novel interactive cognitive architecture based on nonlinear control theory to drive a virtual player (VP) to play the mirror game with a human player (HP) in different configurations. Specifically, we consider two cases: 1) the VP acts as leader and 2) the VP acts as follower. The crucial problem is to design a feedback control architecture capable of imitating and following or leading an HP in a joint action task. The movement of the end-effector of the VP is modeled by means of a feedback controlled Haken-Kelso-Bunz (HKB) oscillator, which is coupled with the observed motion of the HP measured in real time. To this aim, two types of control algorithms (adaptive control and optimal control) are used and implemented on the HKB model so that the VP can generate a human-like motion while satisfying certain kinematic constraints. A proof of convergence of the control algorithms is presented together with an extensive numerical and experimental validation of their effectiveness. A comparison with other existing designs is also discussed, showing the flexibility and the advantages of our control-based approach.

A roadmap to computational social neuroscience

To complement experimental efforts toward understanding human social interactions at both neural and behavioral levels, two computational approaches are presented: (1) a fully parameterizable mathematical model of a social partner, the Human Dynamic Clamp which, by virtue of experimentally controlled interactions between Virtual Partners and real people, allows for emergent behaviors to be studied; and (2) a multiscale neurocomputational model of social coordination that enables exploration of social self-organization at all levels-from neuronal patterns to people interacting with each other. These complementary frameworks and the cross product of their analysis aim at understanding the fundamental principles governing social behavior.

Anisotropy and Antagonism in the Coupling of Two Oscillators: Concepts and Applications for Between-Person Coordination

Coupled oscillators provide a pertinent model approach to study between-person movement dynamics. While ample literature in this respect has considered the influence of external/environmental constraints and/or effects of a difference between the two agents' individual component dynamics (e.g., mismatch in natural frequency), recent studies also started to more directly consider the interaction per-se. The current perspective paper sets forth that while movement coordination dynamics has mainly been studied alongside a model in which the coupling is considered isotropic (i.e., symmetrical; both oscillators coupled to same degree) or strictly unidirectional (e.g., for moving to a given external rhythm), between-agent coupling involves a natural anisotropy: components influence each other bidirectionally to different degrees. Furthermore, recent research from different areas has considered so-called antagonistic or "competitive" coupling, which refers to the idea that one component is positively coupled to the other (attractive interaction), while the coupling in the other direction is negative (repulsive interaction). Although the latter would be rather tricky to address in within-person coordination, it does have strong applications and implications for between-person dynamics, for instance in the study of competitive interactions in sports situations (e.g., attacker-defender) and conflicting social (movement) interactions. The paper concludes by offering a conceptual framework and perspectives for future studies on the dynamic anisotropic nature of the interaction in between-person contexts.

Dynamic similarity promotes interpersonal coordination in joint action

Human movement has been studied for decades, and dynamic laws of motion that are common to all humans have been derived. Yet, every individual moves differently from everyone else (faster/slower, harder/smoother, etc.). We propose here an index of such variability, namely an individual motor signature (IMS) able to capture the subtle differences in the way each of us moves. We show that the IMS of a person is time-invariant and that it significantly differs from those of other individuals. This allows us to quantify the dynamic similarity, a measure of rapport between dynamics of different individuals' movements, and demonstrate that it facilitates coordination during interaction. We use our measure to confirm a key prediction of the theory of similarity that coordination between two individuals performing a joint-action task is higher if their motions share similar dynamic features. Furthermore, we use a virtual avatar driven by an interactive cognitive architecture based on feedback control theory to explore the effects of different kinematic features of the avatar motion on coordination with human players.

Coordination Dynamics in Cognitive Neuroscience

Many researchers and clinicians in cognitive neuroscience hold to a modular view of cognitive function in which the cerebral cortex operates by the activation of areas with circumscribed elementary cognitive functions. Yet an ongoing paradigm shift to a dynamic network perspective is underway. This new viewpoint treats cortical function as arising from the coordination dynamics within and between cortical regions. Cortical coordination dynamics arises due to the unidirectional influences imposed on a cortical area by inputs from other areas that project to it, combined with the projection reciprocity that characterizes cortical connectivity and gives rise to reentrant processing. As a result, cortical dynamics exhibits both segregative and integrative tendencies and gives rise to both cooperative and competitive relations within and between cortical areas that are hypothesized to underlie the emergence of cognition in brains.

Beyond in-phase and anti-phase coordination in a model of joint action

In 1985, Haken, Kelso and Bunz proposed a system of coupled nonlinear oscillators as a model of rhythmic movement patterns in human bimanual coordination. Since then, the Haken-Kelso-Bunz (HKB) model has become a modelling paradigm applied extensively in all areas of movement science, including interpersonal motor coordination. However, all previous studies have followed a line of analysis based on slowly varying amplitudes and rotating wave approximations. These approximations lead to a reduced system, consisting of a single differential equation representing the evolution of the relative phase of the two coupled oscillators: the HKB model of the relative phase. Here we take a different approach and systematically investigate the behaviour of the HKB model in the full four-dimensional state space and for general coupling strengths. We perform detailed numerical bifurcation analyses and reveal that the HKB model supports previously unreported dynamical regimes as well as bistability between a variety of coordination patterns. Furthermore, we identify the stability boundaries of distinct coordination regimes in the model and discuss the applicability of our findings to interpersonal coordination and other joint action tasks.

Entrainment and synchronization in networks of Rayleigh-van der Pol oscillators with diffusive and Haken-Kelso-Bunz couplings

We analyze a network of non-identical Rayleigh-van der Pol (RvdP) oscillators interconnected through either diffusive or nonlinear coupling functions. The work presented here extends existing results on the case of two nonlinearly coupled RvdP oscillators to the problem of considering a network of three or more of them. Specifically, we study synchronization and entrainment in networks of heterogeneous RvdP oscillators and contrast the effects of diffusive linear coupling strategies with the nonlinear Haken-Kelso-Bunz coupling, originally introduced to study human bimanual experiments. We show how convergence of the error among the nodes' trajectories toward a bounded region is possible with both linear and nonlinear coupling functions. Under the assumption that the network is connected, simple, and undirected, analytical results are obtained to prove boundedness of the error when the oscillators are coupled diffusively. All results are illustrated by way of numerical examples and compared with the experimental findings available in the literature on synchronization of people rocking chairs, confirming the effectiveness of the model we propose to capture some of the features of human group synchronization observed experimentally in the previous literature.

Design of a Virtual Player for Joint Improvisation with Humans in the Mirror Game

Joint improvisation is often observed among humans performing joint action tasks. Exploring the underlying cognitive and neural mechanisms behind the emergence of joint improvisation is an open research challenge. This paper investigates jointly improvised movements between two participants in the mirror game, a paradigmatic joint task example. First, experiments involving movement coordination of different dyads of human players are performed in order to build a human benchmark. No designation of leader and follower is given beforehand. We find that joint improvisation is characterized by the lack of a leader and high levels of movement synchronization. Then, a theoretical model is proposed to capture some features of their interaction, and a set of experiments is carried out to test and validate the model ability to reproduce the experimental observations. Furthermore, the model is used to drive a computer avatar able to successfully improvise joint motion with a human participant in real time. Finally, a convergence analysis of the proposed model is carried out to confirm its ability to reproduce joint movements between the participants.

Enhanced emotional responses during social coordination with a virtual partner

Emotion and motion, though seldom studied in tandem, are complementary aspects of social experience. This study investigates variations in emotional responses during movement coordination between a human and a Virtual Partner (VP), an agent whose virtual finger movements are driven by the Haken-Kelso-Bunz (HKB) equations of Coordination Dynamics. Twenty-one subjects were instructed to coordinate finger movements with the VP in either inphase or antiphase patterns. By adjusting model parameters, we manipulated the 'intention' of VP as cooperative or competitive with the human's instructed goal. Skin potential responses (SPR) were recorded to quantify the intensity of emotional response. At the end of each trial, subjects rated the VP's intention and whether they thought their partner was another human being or a machine. We found greater emotional responses when subjects reported that their partner was human and when coordination was stable. That emotional responses are strongly influenced by dynamic features of the VP's behavior, has implications for mental health, brain disorders and the design of socially cooperative machines.

The Virtual Teacher (VT) Paradigm: Learning New Patterns of Interpersonal Coordination Using the Human Dynamic Clamp

The Virtual Teacher paradigm, a version of the Human Dynamic Clamp (HDC), is introduced into studies of learning patterns of inter-personal coordination. Combining mathematical modeling and experimentation, we investigate how the HDC may be used as a Virtual Teacher (VT) to help humans co-produce and internalize new inter-personal coordination pattern(s). Human learners produced rhythmic finger movements whilst observing a computer-driven avatar, animated by dynamic equations stemming from the well-established Haken-Kelso-Bunz (1985) and Schöner-Kelso (1988) models of coordination. We demonstrate that the VT is successful in shifting the pattern co-produced by the VT-human system toward any value (Experiment 1) and that the VT can help humans learn unstable relative phasing patterns (Experiment 2). Using transfer entropy, we find that information flow from one partner to the other increases when VT-human coordination loses stability. This suggests that variable joint performance may actually facilitate interaction, and in the long run learning. VT appears to be a promising tool for exploring basic learning processes involved in social interaction, unraveling the dynamics of information flow between interacting partners, and providing possible rehabilitation opportunities.

Influence of Internal and External Noise on Spontaneous Visuomotor Synchronization

Historically, movement noise or variability is considered to be an undesirable property of biological motor systems. In particular, noise is typically assumed to degrade the emergence and stability of rhythmic motor synchronization. Recently, however, it has been suggested that small levels of noise might actually improve the functioning of motor systems and facilitate their adaptation to environmental events. Here, the authors investigated whether noise can facilitate spontaneous rhythmic visuomotor synchronization. They examined the influence of internal noise in the rhythmic limb movements of participants and external noise in the movement of an oscillating visual stimulus on the occurrence of spontaneous synchronization. By indexing the natural frequency variability of participants and manipulating the frequency variability of the visual stimulus, the authors demonstrated that both internal and external noise degrade synchronization when the participants' and stimulus movement frequencies are similar, but can actually facilitate synchronization when the frequencies are different. Furthermore, the two kinds of noise interact with each other. Internal noise facilitates synchronization only when external noise is minimal and vice versa. Too much internal and external noise together degrades synchronization. These findings open new perspectives for better understanding the role of noise in human rhythmic coordination.

Radical embodied cognitive neuroscience: addressing "grand challenges" of the mind sciences

It is becoming ever more accepted that investigations of mind span the brain, body, and environment. To broaden the scope of what is relevant in such investigations is to increase the amount of data scientists must reckon with. Thus, a major challenge facing scientists who study the mind is how to make big data intelligible both within and between fields. One way to face this challenge is to structure the data within a framework and to make it intelligible by means of a common theory. Radical embodied cognitive neuroscience can function as such a framework, with dynamical systems theory as its methodology, and self-organized criticality as its theory.

The human dynamic clamp as a paradigm for social interaction

Social neuroscience has called for new experimental paradigms aimed toward real-time interactions. A distinctive feature of interactions is mutual information exchange: One member of a pair changes in response to the other while simultaneously producing actions that alter the other. Combining mathematical and neurophysiological methods, we introduce a paradigm called the human dynamic clamp (HDC), to directly manipulate the interaction or coupling between a human and a surrogate constructed to behave like a human. Inspired by the dynamic clamp used so productively in cellular neuroscience, the HDC allows a person to interact in real time with a virtual partner itself driven by well-established models of coordination dynamics. People coordinate hand movements with the visually observed movements of a virtual hand, the parameters of which depend on input from the subject's own movements. We demonstrate that HDC can be extended to cover a broad repertoire of human behavior, including rhythmic and discrete movements, adaptation to changes of pacing, and behavioral skill learning as specified by a virtual "teacher." We propose HDC as a general paradigm, best implemented when empirically verified theoretical or mathematical models have been developed in a particular scientific field. The HDC paradigm is powerful because it provides an opportunity to explore parameter ranges and perturbations that are not easily accessible in ordinary human interactions. The HDC not only enables to test the veracity of theoretical models, it also illuminates features that are not always apparent in real-time human social interactions and the brain correlates thereof.

You are in sync with me: neural correlates of interpersonal synchrony with a partner

Interpersonal synchrony is characterized by a temporary alignment of periodic behaviors with another person. This process requires that at least one of the two individuals monitors and adjusts his/her movements to maintain alignment with the other individual (the referent). Interestingly, recent research on interpersonal synchrony has found that people who are motivated to befriend an unfamiliar social referent tend to automatically synchronize with their social referents, raising the possibility that synchrony may be employed as an affiliation tool. It is unknown, however, whether the opposite is true; that is, whether the person serving as the referent of interpersonal synchrony perceives synchrony with his/her partner or experiences affiliative feelings toward the partner. To address this question, we performed a series of studies on interpersonal synchrony with a total of 100 participants. In all studies, participants served as the referent with no requirement to monitor or align their behavior with their partners. Unbeknown to the participants, the timings of their "partner's" movements were actually determined by a computer program based on the participant's (i.e., referent's) behavior. Overall, our behavioral results showed that the referent of a synchrony task expressed greater perceived synchrony and greater social affiliation toward a synchronous partner (i.e., one displaying low mean asynchrony and/or a narrow asynchrony range) than with an asynchronous partner (i.e., one displaying high mean asynchrony and/or high asynchrony range). Our neuroimaging study extended these results by demonstrating involvement of brain areas implicated in social cognition, embodied cognition, self-other expansion, and action observation as correlates of interpersonal synchrony (vs. asynchrony). These findings have practical implications for social interaction and theoretical implications for understanding interpersonal synchrony and social coordination.

Asymmetric interpersonal coupling in a cyclic sports-related movement task

In interactive sports, teammates and/or opponents mutually tune their behavior. Expert performance thus implies certain interactive abilities, which critically depend on perceptual coupling. To illustrate this assertion, we examined the coordination dynamics with asymmetric interaction of dyads performing a sports-related cyclical movement task. In pairs, basketball players performed lateral defensive slides in in-phase, until a cue prompted them to switch to antiphase coordination. We assessed how these switches were mediated by phase adaptations of each agent under bidirectional (i.e., agents facing one another) and unidirectional (i.e., one agent facing the back of the other) visual interaction conditions. This imposed asymmetry in visual coupling exemplified an imbalance in the interaction (or 'interact-ability') between two agents. The results concurred the asymmetric coupling: during the switch the agent facing the other adapted his phasing more than the other agent. Furthermore, also in the bidirectional condition the coupling revealed dyad-intrinsic asymmetries (e.g., related to implicit follower-leader strategies). Together, this illustrates that interpersonal coordination is characterized by asymmetric coupling between the agents, and highlights how mutual perception of pertinent information mediates interpersonal coordination. This study offered a first step towards analyzing interpersonal coordination dynamics in relation to 'interact-ability'.

Rhythm patterns interaction--synchronization behavior for human-robot joint action

Interactive behavior among humans is governed by the dynamics of movement synchronization in a variety of repetitive tasks. This requires the interaction partners to perform for example rhythmic limb swinging or even goal-directed arm movements. Inspired by that essential feature of human interaction, we present a novel concept and design methodology to synthesize goal-directed synchronization behavior for robotic agents in repetitive joint action tasks. The agents’ tasks are described by closed movement trajectories and interpreted as limit cycles, for which instantaneous phase variables are derived based on oscillator theory. Events segmenting the trajectories into multiple primitives are introduced as anchoring points for enhanced synchronization modes. Utilizing both continuous phases and discrete events in a unifying view, we design a continuous dynamical process synchronizing the derived modes. Inverse to the derivation of phases, we also address the generation of goal-directed movements from the behavioral dynamics. The developed concept is implemented to an anthropomorphic robot. For evaluation of the concept an experiment is designed and conducted in which the robot performs a prototypical pick-and-place task jointly with human partners. The effectiveness of the designed behavior is successfully evidenced by objective measures of phase and event synchronization. Feedback gathered from the participants of our exploratory study suggests a subjectively pleasant sense of interaction created by the interactive behavior. The results highlight potential applications of the synchronization concept both in motor coordination among robotic agents and in enhanced social interaction between humanoid agents and humans.

Sensorimotor synchronization: a review of recent research (2006-2012)

Sensorimotor synchronization (SMS) is the coordination of rhythmic movement with an external rhythm, ranging from finger tapping in time with a metronome to musical ensemble performance. An earlier review (Repp, 2005) covered tapping studies; two additional reviews (Repp, 2006a, b) focused on music performance and on rate limits of SMS, respectively. The present article supplements and extends these earlier reviews by surveying more recent research in what appears to be a burgeoning field. The article comprises four parts, dealing with (1) conventional tapping studies, (2) other forms of moving in synchrony with external rhythms (including dance and nonhuman animals' synchronization abilities), (3) interpersonal synchronization (including musical ensemble performance), and (4) the neuroscience of SMS. It is evident that much new knowledge about SMS has been acquired in the last 7 years.

The metastable brain

Neural ensembles oscillate across a broad range of frequencies and are transiently coupled or "bound" together when people attend to a stimulus, perceive, think, and act. This is a dynamic, self-assembling process, with parts of the brain engaging and disengaging in time. But how is it done? The theory of Coordination Dynamics proposes a mechanism called metastability, a subtle blend of integration and segregation. Tendencies for brain regions to express their individual autonomy and specialized functions (segregation, modularity) coexist with tendencies to couple and coordinate globally for multiple functions (integration). Although metastability has garnered increasing attention, it has yet to be demonstrated and treated within a fully spatiotemporal perspective. Here, we illustrate metastability in continuous neural and behavioral recordings, and we discuss theory and experiments at multiple scales, suggesting that metastable dynamics underlie the real-time coordination necessary for the brain's dynamic cognitive, behavioral, and social functions.

Challenges for the understanding of the dynamics of social coordination

The way people interact can be examined by looking at the way they move relative to each other. Seeking the principles behind those interactions have consequences potentially related to any type of interpersonal function, far beyond the so-called "motor" processes typically associated with the study of movements, be it perceptive, cognitive, affective, pragmatic, or epistemic. Here, we present the way the framework of coordination dynamics define and addresses the interactive actions in a dyad. We first introduce the basics of pattern formation as the roots of the theoretical approach of coordination dynamics, and then the way this framework may contribute to establish a solution to classify behaviors. Thereafter we review promising empirical results on the dynamics of interpersonal coordination, and finally discuss were to go next to decipher the way the coordination between two people and the way each individual contribute may be disentangled.

Leading the follower: an fMRI investigation of dynamic cooperativity and leader-follower strategies in synchronization with an adaptive virtual partner

From everyday experience we know that it is generally easier to interact with someone who adapts to our behavior. Beyond this, achieving a common goal will very much depend on who adapts to whom and to what degree. Therefore, many joint action tasks such as musical performance prove to be more successful when defined leader-follower roles are established. In the present study, we present a novel approach to explore the mechanisms of how individuals lead and, using functional magnetic resonance imaging (fMRI), probe the neural correlates of leading. Specifically, we implemented an adaptive virtual partner (VP), an auditory pacing signal, with which individuals were instructed to tap in synchrony while maintaining a steady tempo. By varying the degree of temporal adaptation (period correction) implemented by the VP, we manipulated the objective control individuals had to exert to maintain the overall tempo of the pacing sequence (which was prone to tempo drift with high levels of period correction). Our imaging data revealed that perceiving greater influence and leading are correlated with right lateralized frontal activation of areas involved in cognitive control and self-related processing. Using participants' subjective ratings of influence and task difficulty, we classified a subgroup of our cohort as "leaders", individuals who found the task of synchronizing easier when they felt more in control. Behavioral tapping measures showed that leaders employed less error correction and focused more on self-tapping (prioritizing the instruction to maintain the given tempo) than on the stability of the interaction (prioritizing the instruction to synchronize with the VP), with correlated activity in areas involved in self-initiated action including the pre-supplementary motor area.

Cyborg psychiatry to ensure agency and autonomy in mental disorders. A proposal for neuromodulation therapeutics

Neuromodulation therapeutics—as repeated Transcranial Magnetic Stimulation (rTMS) and neurofeedback—are valuable tools for psychiatry. Nevertheless, they currently face some limitations: rTMS has confounding effects on neural activation patterns, and neurofeedback fails to change neural dynamics in some cases. Here we propose how coupling rTMS and neurofeedback can tackle both issues by adapting neural activations during rTMS and actively guiding individuals during neurofeedback. An algorithmic challenge then consists in designing the proper recording, processing, feedback, and control of unwanted effects. But this new neuromodulation technique also poses an ethical challenge: ensuring treatment occurs within a biopsychosocial model of medicine, while considering both the interaction between the patients and the psychiatrist, and the maintenance of individuals' autonomy. Our solution is the concept of Cyborg psychiatry, which embodies the technique and includes a self-engaged interaction between patients and the neuromodulation device.

The situated HKB model: how sensorimotor spatial coupling can alter oscillatory brain dynamics

Despite the increase of both dynamic and embodied/situated approaches in cognitive science, there is still little research on how coordination dynamics under a closed sensorimotor loop might induce qualitatively different patterns of neural oscillations compared to those found in isolated systems. We take as a departure point the Haken-Kelso-Bunz (HKB) model, a generic model for dynamic coordination between two oscillatory components, which has proven useful for a vast range of applications in cognitive science and whose dynamical properties are well understood. In order to explore the properties of this model under closed sensorimotor conditions we present what we call the situated HKB model: a robotic model that performs a gradient climbing task and whose "brain" is modeled by the HKB equation. We solve the differential equations that define the agent-environment coupling for increasing values of the agent's sensitivity (sensor gain), finding different behavioral strategies. These results are compared with two different models: a decoupled HKB with no sensory input and a passively-coupled HKB that is also decoupled but receives a structured input generated by a situated agent. We can precisely quantify and qualitatively describe how the properties of the system, when studied in coupled conditions, radically change in a manner that cannot be deduced from the decoupled HKB models alone. We also present the notion of neurodynamic signature as the dynamic pattern that correlates with a specific behavior and we show how only a situated agent can display this signature compared to an agent that simply receives the exact same sensory input. To our knowledge, this is the first analytical solution of the HKB equation in a sensorimotor loop and qualitative and quantitative analytic comparison of spatially coupled vs. decoupled oscillatory controllers. Finally, we discuss the limitations and possible generalization of our model to contemporary neuroscience and philosophy of mind.