Interactive brains, social minds: Neural and physiological mechanisms of interpersonal action coordination

Hyper-brain hyper-frequency network topology dynamics when playing guitar in quartet

Ensemble music performance is a highly coordinated form of social behavior requiring not only precise motor actions but also synchronization of different neural processes both within and between the brains of ensemble players. In previous analyses, which were restricted to within-frequency coupling (WFC), we showed that different frequencies participate in intra- and inter-brain coordination, exhibiting distinct network topology dynamics that underlie coordinated actions and interactions. However, many of the couplings both within and between brains are likely to operate across frequencies. Hence, to obtain a more complete picture of hyper-brain interaction when musicians play the guitar in a quartet, cross-frequency coupling (CFC) has to be considered as well. Furthermore, WFC and CFC can be used to construct hyper-brain hyper-frequency networks (HB-HFNs) integrating all the information flows between different oscillation frequencies, providing important details about ensemble interaction in terms of network topology dynamics (NTD). Here, we reanalyzed EEG (electroencephalogram) data obtained from four guitarists playing together in quartet to explore changes in HB-HFN topology dynamics and their relation to acoustic signals of the music. Our findings demonstrate that low-frequency oscillations (e.g., delta, theta, and alpha) play an integrative or pacemaker role in such complex networks and that HFN topology dynamics are specifically related to the guitar quartet playing dynamics assessed by sound properties. Simulations by link removal showed that the HB-HFN is relatively robust against loss of connections, especially when the strongest connections are preserved and when the loss of connections only affects the brain of one guitarist. We conclude that HB-HFNs capture neural mechanisms that support interpersonally coordinated action and behavioral synchrony.

A practical guide to EEG hyperscanning in joint action research: from motivation to implementation

Developments in cognitive neuroscience have led to the emergence of hyperscanning, the simultaneous measurement of brain activity from multiple people. Hyperscanning is useful for investigating social cognition, including joint action, because of its ability to capture neural processes that occur within and between people as they coordinate actions toward a shared goal. Here, we provide a practical guide for researchers considering using hyperscanning to study joint action and seeking to avoid frequently raised concerns from hyperscanning skeptics. We focus specifically on Electroencephalography (EEG) hyperscanning, which is widely available and optimally suited for capturing fine-grained temporal dynamics of action coordination. Our guidelines cover questions that are likely to arise when planning a hyperscanning project, ranging from whether hyperscanning is appropriate for answering one's research questions to considerations for study design, dependent variable selection, data analysis and visualization. By following clear guidelines that facilitate careful consideration of the theoretical implications of research design choices and other methodological decisions, joint action researchers can mitigate interpretability issues and maximize the benefits of hyperscanning paradigms.

Leadership dynamics in musical groups: Quantifying effects of musical structure on directionality of influence in concert performance videos

Music ensemble performance provides an ecologically valid context for investigating leadership dynamics in small group interactions. Musical texture, specifically the relative salience of simultaneously sounding ensemble parts, is a feature that can potentially alter leadership dynamics by introducing hierarchical relationships between individual parts. The present study extended previous work on quantifying interpersonal coupling in musical ensembles by examining the relationship between musical texture and leader-follower relations, operationalised as directionality of influence between co-performers' body motion in concert video recordings. It was hypothesised that the directionality of influence, indexed by Granger Causality, would be greater for 'homophonic' textures with a clear distinction between melody and accompaniment parts than for 'polyphonic' textures with less distinction between melody and accompaniment. This hypothesis was tested by using pose estimation algorithms to track instrumentalists' body movements in a string quartet and a clarinet quintet, and then applying Granger Causality analysis to their head motion to estimate directional influence between instrumentalist pairs for sections of the pieces that varied in texture. It was found that Granger Causality values were generally higher (indicating greater directionality of influence) for homophonic than polyphonic textures. Furthermore, considering melody and accompaniment instrument roles revealed more evidence for the melody instrument influencing accompanying instruments than vice versa, plus a high degree of directionality among accompanying instruments, in homophonic textures. These observed patterns of directional information flow in co-performer body motion are consistent with changing leader-follower relations depending on hierarchical relations between ensemble parts in terms of the relative salience of melodic material in the musical texture. The finding that automatic pose estimation can detect modulations of leadership dynamics in standard video recordings under naturalistic performance conditions has implications for investigating interpersonal coordination in large-scale music video datasets representing different cultural traditions, and for exploring nonverbal communication in group activities more generally.

Coordination as inference in multi-agent reinforcement learning

The Centralized Training and Decentralized Execution (CTDE) paradigm, where a centralized critic is allowed to access global information during the training phase while maintaining the learned policies executed with only local information in a decentralized way, has achieved great progress in recent years. Despite the progress, CTDE may suffer from the issue of Centralized-Decentralized Mismatch (CDM): the suboptimality of one agent's policy can exacerbate policy learning of other agents through the centralized joint critic. In contrast to centralized learning, the cooperative model that most closely resembles the way humans cooperate in nature is fully decentralized, i.e. Independent Learning (IL). However, there are still two issues that need to be addressed before agents coordinate through IL: (1) how agents are aware of the presence of other agents, and (2) how to coordinate with other agents to improve joint policy under IL. In this paper, we propose an inference-based coordinated MARL method: Deep Motor System (DMS). DMS first presents the idea of individual intention inference where agents are allowed to disentangle other agents from their environment. Secondly, causal inference was introduced to enhance coordination by reasoning each agent's effect on others' behavior. The proposed model was extensively experimented on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that the proposed method outperforms independent learning algorithms and the coordination behavior among agents can be learned even without the CTDE paradigm compared to the state-of-the-art baselines including IPPO and HAPPO.

An ecological study protocol for the multimodal investigation of the neurophysiological underpinnings of dyadic joint action

A novel multimodal experimental setup and dyadic study protocol were designed to investigate the neurophysiological underpinnings of joint action through the synchronous acquisition of EEG, ECG, EMG, respiration and kinematic data from two individuals engaged in ecologic and naturalistic cooperative and competitive joint actions involving face-to-face real-time and real-space coordinated full body movements. Such studies are still missing because of difficulties encountered in recording reliable neurophysiological signals during gross body movements, in synchronizing multiple devices, and in defining suitable study protocols. The multimodal experimental setup includes the synchronous recording of EEG, ECG, EMG, respiration and kinematic signals of both individuals via two EEG amplifiers and a motion capture system that are synchronized via a single-board microcomputer and custom Python scripts. EEG is recorded using new dry sports electrode caps. The novel study protocol is designed to best exploit the multimodal data acquisitions. Table tennis is the dyadic motor task: it allows naturalistic and face-to-face interpersonal interactions, free in-time and in-space full body movement coordination, cooperative and competitive joint actions, and two task difficulty levels to mimic changing external conditions. Recording conditions-including minimum table tennis rally duration, sampling rate of kinematic data, total duration of neurophysiological recordings-were defined according to the requirements of a multilevel analytical approach including a neural level (hyperbrain functional connectivity, Graph Theoretical measures and Microstate analysis), a cognitive-behavioral level (integrated analysis of neural and kinematic data), and a social level (extending Network Physiology to neurophysiological data recorded from two interacting individuals). Four practical tests for table tennis skills were defined to select the study population, permitting to skill-match the dyad members and to form two groups of higher and lower skilled dyads to explore the influence of skill level on joint action performance. Psychometric instruments are included to assess personality traits and support interpretation of results. Studying joint action with our proposed protocol can advance the understanding of the neurophysiological mechanisms sustaining daily life joint actions and could help defining systems to predict cooperative or competitive behaviors before being overtly expressed, particularly useful in real-life contexts where social behavior is a main feature.

Cognitive Reappraisal and Expressive Suppression Evoke Distinct Neural Connections during Interpersonal Emotion Regulation

Interpersonal emotion regulation is the dynamic process where the regulator aims to change the target's emotional state, which is presumed to engage three neural systems: cognitive control (i.e., dorsal and ventral lateral PFC, etc.), empathy/social cognition (i.e., dorsal premotor regions, temporal-parietal junction, etc.), and affective response (i.e., insula, amygdala, etc.). This study aimed to identify the underlying neural correlate (especially the interpersonal one), of interpersonal emotion regulation based on two typical strategies (cognitive appraisal, expressive suppression). Thirty-four female dyads (friends) were randomly assigned into two strategy groups, with one assigned as the target and the other as the regulator to downregulate the target's negative emotions using two strategies. A functional near-infrared spectroscopy system was used to simultaneously measure participants' neural activity. Results showed that these two strategies could successfully downregulate the targets' negative emotions. Both strategies evoked intrapersonal and interpersonal neural couplings between the cognitive control, social cognition, and mirror neuron systems (e.g., PFC, temporal-parietal junction, premotor cortex, etc.), whereas cognitive reappraisal (vs expressive suppression) evoked a broader pattern. Further, cognitive reappraisal involved increased interpersonal brain synchronization between the prefrontal and temporal areas at the sharing stage, whereas expressive suppression evoked increased interpersonal brain synchronization associated with the PFC at the regulation stage. These findings indicate that intrapersonal and interpersonal neural couplings associated with regions within the abovementioned systems, possibly involving mental processes, such as cognitive control, mentalizing, and observing, underlie interpersonal emotion regulation based on cognitive reappraisal or expressive suppression.SIGNIFICANCE STATEMENT As significant as intrapersonal emotion regulation, interpersonal emotion regulation subserves parent-child, couple, and leader-follower relationships. Despite enormous growth in research on intrapersonal emotion regulation, the field lacks insight into the neural correlates underpinning interpersonal emotion regulation. This study aimed to probe the underlying neural correlates of interpersonal emotion regulation using a multibrain neuroimaging (i.e., hyperscanning) based on functional near-infrared spectroscopy. Results showed that both cognitive reappraisal and expressive suppression strategies successfully downregulated the target's negative emotions. More importantly, they evoked intrapersonal and interpersonal neural couplings associated with regions within the cognitive control, social cognition, and mirror neuron systems, possibly involving mental processes, such as cognitive control, mentalizing, and observing. These findings deepen our understanding of the neural correlates underpinning interpersonal emotion regulation.

Dynamics in interbrain synchronization while playing a piano duet

Humans interact with each other through actions that are implemented by sensory and motor processes. To investigate the role of interbrain synchronization emerging during interpersonal action coordination, electroencephalography data from 13 pairs of pianists were recorded simultaneously while they performed a duet together. The study aimed to investigate whether interbrain phase couplings can be reduced to similar bottom-up driven processes during synchronous play, or rather represent cognitive top-down control required during periods of higher coordination demands. To induce such periods, one of the musicians acted as a confederate who deliberately desynchronized the play. As intended, on the behavioral level, the perturbation caused a breakdown in the synchronization of the musicians' play and in its stability across trials. On the brain level, interbrain synchrony, as measured by the interbrain phase coherence (IPC), increased in the delta and theta frequency bands during perturbation as compared to non-perturbed trials. Interestingly, this increase in IPC in the delta band was accompanied by the shift of the phase difference angle from in-phase toward anti-phase synchrony. In conclusion, the current study demonstrates that interbrain synchronization is based on the interpersonal temporal alignment of different brain mechanisms and is not simply reducible to similar sensory or motor responses.

Quantification of inter-brain coupling: A review of current methods used in haemodynamic and electrophysiological hyperscanning studies

Hyperscanning is a form of neuroimaging experiment where the brains of two or more participants are imaged simultaneously whilst they interact. Within the domain of social neuroscience, hyperscanning is increasingly used to measure inter-brain coupling (IBC) and explore how brain responses change in tandem during social interaction. In addition to cognitive research, some have suggested that quantification of the interplay between interacting participants can be used as a biomarker for a variety of cognitive mechanisms aswell as to investigate mental health and developmental conditions including schizophrenia, social anxiety and autism. However, many different methods have been used to quantify brain coupling and this can lead to questions about comparability across studies and reduce research reproducibility. Here, we review methods for quantifying IBC, and suggest some ways moving forward. Following the PRISMA guidelines, we reviewed 215 hyperscanning studies, across four different brain imaging modalities: functional near-infrared spectroscopy (fNIRS), functional magnetic resonance (fMRI), electroencephalography (EEG) and magnetoencephalography (MEG). Overall, the review identified a total of 27 different methods used to compute IBC. The most common hyperscanning modality is fNIRS, used by 119 studies, 89 of which adopted wavelet coherence. Based on the results of this literature survey, we first report summary statistics of the hyperscanning field, followed by a brief overview of each signal that is obtained from each neuroimaging modality used in hyperscanning. We then discuss the rationale, assumptions and suitability of each method to different modalities which can be used to investigate IBC. Finally, we discuss issues surrounding the interpretation of each method.

Intra- and inter-brain coupling and activity dynamics during improvisational music therapy with a person with dementia: an explorative EEG-hyperscanning single case study

Objective

Real-life research into the underlying neural dynamics of improvisational music therapy, used with various clinical populations, is largely lacking. This single case study explored within-session differences in musical features and in within- and between-brain coupling between a Person with Dementia (PwD) and a music therapist during a music therapy session.

Methods

Dual-EEG from a music therapist and a PwD (male, 31 years) was recorded. Note density, pulse clarity and synchronicity were extracted from audio-visual data. Three music therapists identified moments of interest and no interest (MOI/MONI) in two drum improvisations. The Integrative Coupling Index, reflecting time-lagged neural synchronization, and musical features were compared between the MOI and MONI.

Results

Between-brain coupling of 2 Hz activity was increased during the MOI, showing anteriority of the therapist's neural activity. Within-brain coupling for the PwD was stronger from frontal and central areas during the MOI, but within-brain coupling for the therapist was stronger during MONI. Differences in musical features indicated that both acted musically more similar to one another during the MOI.

Conclusion

Within-session differences in neural synchronization and musical features highlight the dynamic nature of music therapy.

Significance

The findings contribute to a better understanding of social and affective processes in the brain and (interactive) musical behaviors during specific moments in a real-life music therapy session. This may provide insights into the role of such moments for relational-therapeutic processes.

Cortico-cerebellar audio-motor regions coordinate self and other in musical joint action

Joint music performance requires flexible sensorimotor coordination between self and other. Cognitive and sensory parameters of joint action-such as shared knowledge or temporal (a)synchrony-influence this coordination by shifting the balance between self-other segregation and integration. To investigate the neural bases of these parameters and their interaction during joint action, we asked pianists to play on an MR-compatible piano, in duet with a partner outside of the scanner room. Motor knowledge of the partner's musical part and the temporal compatibility of the partner's action feedback were manipulated. First, we found stronger activity and functional connectivity within cortico-cerebellar audio-motor networks when pianists had practiced their partner's part before. This indicates that they simulated and anticipated the auditory feedback of the partner by virtue of an internal model. Second, we observed stronger cerebellar activity and reduced behavioral adaptation when pianists encountered subtle asynchronies between these model-based anticipations and the perceived sensory outcome of (familiar) partner actions, indicating a shift towards self-other segregation. These combined findings demonstrate that cortico-cerebellar audio-motor networks link motor knowledge and other-produced sounds depending on cognitive and sensory factors of the joint performance, and play a crucial role in balancing self-other integration and segregation.

Brain networks for temporal adaptation, anticipation, and sensory-motor integration in rhythmic human behavior

Human interaction often requires the precise yet flexible interpersonal coordination of rhythmic behavior, as in group music making. The present fMRI study investigates the functional brain networks that may facilitate such behavior by enabling temporal adaptation (error correction), prediction, and the monitoring and integration of information about 'self' and the external environment. Participants were required to synchronize finger taps with computer-controlled auditory sequences that were presented either at a globally steady tempo with local adaptations to the participants' tap timing (Virtual Partner task) or with gradual tempo accelerations and decelerations but without adaptation (Tempo Change task). Connectome-based predictive modelling was used to examine patterns of brain functional connectivity related to individual differences in behavioral performance and parameter estimates from the adaptation and anticipation model (ADAM) of sensorimotor synchronization for these two tasks under conditions of varying cognitive load. Results revealed distinct but overlapping brain networks associated with ADAM-derived estimates of temporal adaptation, anticipation, and the integration of self-controlled and externally controlled processes across task conditions. The partial overlap between ADAM networks suggests common hub regions that modulate functional connectivity within and between the brain's resting-state networks and additional sensory-motor regions and subcortical structures in a manner reflecting coordination skill. Such network reconfiguration might facilitate sensorimotor synchronization by enabling shifts in focus on internal and external information, and, in social contexts requiring interpersonal coordination, variations in the degree of simultaneous integration and segregation of these information sources in internal models that support self, other, and joint action planning and prediction.

Interpersonal synchrony when singing in a choir

Singing in a choir has long been known to enhance well-being and protect mental health. Clearly, the experience of a uniquely harmonious social activity is very satisfying for the singers. How might this come about? One of the important factors positively associated with well-being is interpersonal action coordination allowing the choir to function as a whole. This review focuses on temporal coordination dynamics of physiological systems and/or subsystems forming part or the core of the functional substrate of choir singing. These coordination dynamics will be evaluated with respect to the concept of a superordinate system, or superorganism, based on the principles of self-organization and circular causality. We conclude that choral singing is a dynamic process requiring tight interpersonal action coordination that is characterized by coupled physiological systems and specific network topology dynamics, representing a potent biomarker for social interaction.

How to orchestrate a soccer team: Generalized synchronization promoted by rhythmic acoustic stimuli

Interpersonal coordination requires precise actions concerted in space and time in a self-organized manner. We found, using soccer teams as a testing ground, that a common timeframe provided by adequate acoustic stimuli improves the interplay between teammates. We provide quantitative evidence that the connectivity between teammates and the scoring rate of male soccer teams improve significantly when playing under the influence of an appropriate acoustic environment. Unexpectedly, female teams do not show any improvement under the same experimental conditions. We show by follow-up experiments that the acoustic rhythm modulates the attention level of the participants with a pronounced tempo preference and a marked gender difference in the preferred tempo. These results lead to a consistent explanation in terms of the dynamical system theory, nonlinear resonances, and dynamic attention theory, which may illuminate generic mechanisms of the brain dynamics and may have an impact on the design of novel training strategies in team sports.

A Neural Sensor with a Nanocomposite Interface for the Study of Spike Characteristics of Hippocampal Neurons under Learning Training

Both the cellular- and population-level properties of involved neurons are essential for unveiling the learning and memory functions of the brain. To give equal attention to these two aspects, neural sensors based on microelectrode arrays (MEAs) have been in the limelight due to their noninvasive detection and regulation capabilities. Here, we fabricated a neural sensor using carboxylated graphene/3,4-ethylenedioxythiophene:polystyrenesulfonate (cGO/PEDOT:PSS), which is effective in sensing and monitoring neuronal electrophysiological activity in vitro for a long time. The cGO/PEDOT:PSS-modified microelectrodes exhibited a lower electrochemical impedance (7.26 ± 0.29 kΩ), higher charge storage capacity (7.53 ± 0.34 mC/cm²), and improved charge injection (3.11 ± 0.25 mC/cm²). In addition, their performance was maintained after 2 to 4 weeks of long-term cell culture and 50,000 stimulation pulses. During neural network training, the sensors were able to induce learning function in hippocampal neurons through precise electrical stimulation and simultaneously detect changes in neural activity at multiple levels. At the cellular level, not only were three kinds of transient responses to electrical stimulation sensed, but electrical stimulation was also found to affect inhibitory neurons more than excitatory neurons. As for the population level, changes in connectivity and firing synchrony were identified. The cGO/PEDOT:PSS-based neural sensor offers an excellent tool in brain function development and neurological disease treatment.

Audience Interbrain Synchrony During Live Music Is Shaped by Both the Number of People Sharing Pleasure and the Strength of This Pleasure

The study of interbrain coupling in a group of people attending a concert together is a favorable framework to estimate group emotions and more precisely emotional connection between people sharing situations in the same environment. It offers the advantage of studying interactions at the group level. By recording the cerebral activity of people from an audience during a concert using electroencephalography, we previously demonstrated that the higher the emotions and the physically closer the people were, the more the interbrain synchrony (IBS) was enhanced. To further investigate the parameters that shaped inter-brain synchronization in this context, we now focus on the emotional dynamics of the group as a whole by identifying specific moments in the concert that evoked strong or weak emotions, as well as strong or weak emotional cohesion between individuals. We demonstrated that audience interbrain synchrony is mainly associated with experiencing high musical pleasure and that the group emotional cohesion can enhance IBS, but alone is not the major parameter that shapes it in this context.

Neural Synchrony and Network Dynamics in Social Interaction: A Hyper-Brain Cell Assembly Hypothesis

Mounting neurophysiological evidence suggests that interpersonal interaction relies on continual communication between cell assemblies within interacting brains and continual adjustments of these neuronal dynamic states between the brains. In this Hypothesis and Theory article, a Hyper-Brain Cell Assembly Hypothesis is suggested on the basis of a conceptual review of neural synchrony and network dynamics and their roles in emerging cell assemblies within the interacting brains. The proposed hypothesis states that such cell assemblies can emerge not only within, but also between the interacting brains. More precisely, the hyper-brain cell assembly encompasses and integrates oscillatory activity within and between brains, and represents a common hyper-brain unit, which has a certain relation to social behavior and interaction. Hyper-brain modules or communities, comprising nodes across two or several brains, are considered as one of the possible representations of the hypothesized hyper-brain cell assemblies, which can also have a multidimensional or multilayer structure. It is concluded that the neuronal dynamics during interpersonal interaction is brain-wide, i.e., it is based on common neuronal activity of several brains or, more generally, of the coupled physiological systems including brains.

EEG Hyperscanning and Qualitative Analysis of Moments of Interest in Music Therapy for Stroke Rehabilitation—A Feasibility Study

Interdisciplinary research into the underlying neural processes of music therapy (MT) and subjective experiences of patients and therapists are largely lacking. The aim of the current study was to assess the feasibility of newly developed procedures (including electroencephalography/electrocardiography hyperscanning, synchronous audio–video monitoring, and qualitative interviews) to study the personal experiences and neuronal dynamics of moments of interest during MT with stroke survivors. The feasibility of our mobile setup and procedures as well as their clinical implementation in a rehabilitation centre and an acute hospital ward were tested with four phase C patients. Protocols and interviews were used for the documentation and analysis of the feasibility. Recruiting patients for MT sessions was feasible, although data collection on three consecutive weeks was not always possible due to organisational constraints, especially in the hospital with acute ward routines. Research procedures were successfully implemented, and according to interviews, none of the patients reported any burden, tiredness, or increased stress due to the research procedures, which lasted approx. 3 h (ranging from 135 min to 209 min) for each patient. Implementing the research procedures in a rehabilitation unit with stroke patients was feasible, and only small adaptations were made for further research.

Probing associations between interbrain synchronization and interpersonal action coordination during guitar playing

Playing music in an ensemble is a highly coordinated human action involving musicians' brain-body interactions at different levels of physical and cortical organization in time and space. It has been suggested that interbrain phase synchronization plays an essential role in musical interaction. In this study, we aimed to explore associations between interbrain synchronization and interpersonal action coordination, using electroencephalographic recordings of the brain activity of guitarists playing in a duet as well as the acoustic recordings of their music. By applying phase synchronization algorithms to the musicians' brain activities and the sounds produced during guitar playing, we show that synchronous brain activity is strongly related to instrument sounds and behavioral play-onset synchrony, as indicated by phase alignment in relation to the time differences in play onsets and an angular-linear correlation between phase and time differences across trials and guitarist pairs. Interestingly, this correlation was especially strong in the first part of the music piece, when the guitarists seem to adjust their coordinated play onsets and brain rhythms actively. This suggests that the methods capturing intra- and interbrain synchronization and its relations to coordinated playing provide crucial information about the underlying mechanisms of interpersonal action coordination.