Joint Action, Interactive Alignment, and Dialog

Different effects of visual occlusion on interpersonal coordination of head and body movements during dyadic conversations

Introduction

In recent decades, interpersonal coordination and synchrony have been extensively examined in the field of psychology and cognitive science. Studies suggest that perceptual information enables interpersonal coordination and that perceptual noise may even enhance coordination. However, how these perceptual factors influence interpersonal coordination dynamics between head and body movements remains unclear. This study investigated the effect of visual information on the interpersonal coordination of head and body movements during dyadic conversations.

Methods

The availability of visual information was manipulated by positioning a partition halfway between a pair of participants, and the conversations were recorded using a video camera. A video-based human pose estimation software (OpenPose) was used to quantify each interlocutor's head and body movements, which were submitted for cross-recurrence quantification analysis (CRQA), to assess the degree of coordination between the interlocutors.

Results

The results showed different effects between head- and body-movement coordination (i.e., a CRQA measure, maximum line length). The occlusion of visual information increased head-movement coordination, whereas it decreased body-movement coordination.

Discussion

The results suggest that a distinct mechanism may be present at the head- and body-movement coordination level and this study observed differing appearances of compensatory behaviors. Further studies should be conducted to investigate the complex relationships between interpersonal coordination dynamics and various kinds of communication constraints, such as long-term or short-term, and lower-order (perceptual-motor) or higher-order (cognitive-social) level constraints.

Prospects for Augmenting Team Interactions with Real-Time Coordination-Based Measures in Human-Autonomy Teams

Complex work in teams requires coordination across team members and their technology as well as the ability to change and adapt over time to achieve effective performance. To support such complex interactions, recent efforts have worked toward the design of adaptive human-autonomy teaming systems that can provide feedback in or near real time to achieve the desired individual or team results. However, while significant advancements have been made to better model and understand the dynamics of team interaction and its relationship with task performance, appropriate measures of team coordination and computational methods to detect changes in coordination have not yet been widely investigated. Having the capacity to measure coordination in real time is quite promising as it provides the opportunity to provide adaptive feedback that may influence and regulate teams' coordination patterns and, ultimately, drive effective team performance. A critical requirement to reach this potential is having the theoretical and empirical foundation from which to do so. Therefore, the first goal of the paper is to review approaches to coordination dynamics, identify current research gaps, and draw insights from other areas, such as social interaction, relationship science, and psychotherapy. The second goal is to collate extant work on feedback and advance ideas for adaptive feedback systems that have potential to influence coordination in a way that can enhance the effectiveness of team interactions. In addressing these two goals, this work lays the foundation as well as plans for the future of human-autonomy teams that augment team interactions using coordination-based measures.

Dyadic body competence predicts movement synchrony during the mirror game

The process of synchronizing our body movements with others is known to enhance rapport, affect, and prosociality. Furthermore, emerging evidence suggests that synchronizing activities may enhance cognitive performance. Unknown, by contrast, is the extent to which people's individual traits and experiences influence their ability to achieve and maintain movement synchrony with another person, which is key for unlocking the social and affective benefits of movement synchrony. Here, we take a dyad-centered approach to gain a deeper understanding of the role of embodiment in achieving and maintaining movement synchrony. Using existing data, we explored the relationship between body competence and body perception scores at the level of the dyad, and the dyad's movement synchrony and complexity while playing a 2.5-min movement mirroring game. The data revealed that dyadic body competence scores positively correlate with movement synchrony, but not complexity, and that dyadic body perception scores are not associated with movement synchrony or complexity. Movement synchrony was greater when the more experienced member of the dyad was responsible for copying movements. Finally, movement synchrony and complexity were stable across the duration of the mirror game. These findings show that movement synchrony is sensitive to the composition of the dyad involved, specifically the dyad's embodiment, illuminating the value of dyadic approaches to understanding body movements in social contexts.

Objective Linguistic Markers Associated with Callous-Unemotional Traits in Early Childhood

BACKGROUND

Callous-unemotional (CU) traits are associated with interpersonal difficulties and risk for severe conduct problems (CP). The ability to communicate thoughts and feelings is critical to social success, with language a promising treatment target. However, no prior studies have examined objective linguistic correlates of childhood CU traits in early childhood, which could give insight into underlying risk mechanisms and novel target treatments.

METHODS

We computed lexical (positive emotion, sad, and anger words) and conversational (interruptions and speech rate) markers produced by 131 children aged 5-6 years (M = 5.98; SD = 0.54, 58.8% female) and their parents while narrating wordless storybooks during two online visits separated by 6-8 weeks (M = 6.56, SD = 1.11; two books, order counterbalanced). Audio recordings were diarized, time-aligned, and orthographically transcribed using WebTrans. Conversational markers were calculated using R and word frequencies were calculated using Linguistic Inquiry and Word Count (LIWC) software. We examined links between child CU traits and linguistic markers, and explored whether relationships were moderated by child sex.

RESULTS

Higher CU traits were associated with fewer positive emotion words produced by parents and children. Higher CU traits were also associated with greater concordance in the degree of interruptions and expression of anger emotion words by parents and children.

CONCLUSIONS

Results suggest that objective linguistic correlates of CU traits are detectable during early childhood, which could inform adjunctive treatment modules that improve outcomes by precisely tracking and targeting subtle communication patterns.

Consensus-building conversation leads to neural alignment

Conversation is a primary means of social influence, but its effects on brain activity remain unknown. Previous work on conversation and social influence has emphasized public compliance, largely setting private beliefs aside. Here, we show that consensus-building conversation aligns future brain activity within groups, with alignment persisting through novel experiences participants did not discuss. Participants watched ambiguous movie clips during fMRI scanning, then conversed in groups with the goal of coming to a consensus about each clip's narrative. After conversation, participants' brains were scanned while viewing the clips again, along with novel clips from the same movies. Groups that reached consensus showed greater similarity of brain activity after conversation. Participants perceived as having high social status spoke more and signaled disbelief in others, and their groups had unequal turn-taking and lower neural alignment. By contrast, participants with central positions in their real-world social networks encouraged others to speak, facilitating greater group neural alignment. Socially central participants were also more likely to become neurally aligned to others in their groups.

Federated inference and belief sharing

This paper concerns the distributed intelligence or federated inference that emerges under belief-sharing among agents who share a common world-and world model. Imagine, for example, several animals keeping a lookout for predators. Their collective surveillance rests upon being able to communicate their beliefs-about what they see-among themselves. But, how is this possible? Here, we show how all the necessary components arise from minimising free energy. We use numerical studies to simulate the generation, acquisition and emergence of language in synthetic agents. Specifically, we consider inference, learning and selection as minimising the variational free energy of posterior (i.e., Bayesian) beliefs about the states, parameters and structure of generative models, respectively. The common theme-that attends these optimisation processes-is the selection of actions that minimise expected free energy, leading to active inference, learning and model selection (a.k.a., structure learning). We first illustrate the role of communication in resolving uncertainty about the latent states of a partially observed world, on which agents have complementary perspectives. We then consider the acquisition of the requisite language-entailed by a likelihood mapping from an agent's beliefs to their overt expression (e.g., speech)-showing that language can be transmitted across generations by active learning. Finally, we show that language is an emergent property of free energy minimisation, when agents operate within the same econiche. We conclude with a discussion of various perspectives on these phenomena; ranging from cultural niche construction, through federated learning, to the emergence of complexity in ensembles of self-organising systems.

Is song processing distinct and special in the auditory cortex?

Is the singing voice processed distinctively in the human brain? In this Perspective, we discuss what might distinguish song processing from speech processing in light of recent work suggesting that some cortical neuronal populations respond selectively to song and we outline the implications for our understanding of auditory processing. We review the literature regarding the neural and physiological mechanisms of song production and perception and show that this provides evidence for key differences between song and speech processing. We conclude by discussing the significance of the notion that song processing is special in terms of how this might contribute to theories of the neurobiological origins of vocal communication and to our understanding of the neural circuitry underlying sound processing in the human cortex.

Interactionally Embedded Gestalt Principles of Multimodal Human Communication

Natural human interaction requires us to produce and process many different signals, including speech, hand and head gestures, and facial expressions. These communicative signals, which occur in a variety of temporal relations with each other (e.g., parallel or temporally misaligned), must be rapidly processed as a coherent message by the receiver. In this contribution, we introduce the notion of interactionally embedded, affordance-driven gestalt perception as a framework that can explain how this rapid processing of multimodal signals is achieved as efficiently as it is. We discuss empirical evidence showing how basic principles of gestalt perception can explain some aspects of unimodal phenomena such as verbal language processing and visual scene perception but require additional features to explain multimodal human communication. We propose a framework in which high-level gestalt predictions are continuously updated by incoming sensory input, such as unfolding speech and visual signals. We outline the constituent processes that shape high-level gestalt perception and their role in perceiving relevance and prägnanz. Finally, we provide testable predictions that arise from this multimodal interactionally embedded gestalt-perception framework. This review and framework therefore provide a theoretically motivated account of how we may understand the highly complex, multimodal behaviors inherent in natural social interaction.

Shibboleth: An agent-based model of signalling mimicry

Mimicry is an essential strategy for exploiting competitors in competitive co-evolutionary relationships. Protection against mimicry may, furthermore, be a driving force in human linguistic diversity: the potential harm caused by failing to detect mimicked group-identity signals may select for high sensitivity to mimicry of honest group members. Here we describe the results of five agent-based models that simulate multi-generational interactions between two groups of individuals: original members of a group with an honest identity signal, and members of an outsider group who mimic that signal, aiming to pass as members of the in-group. The models correspond to the Biblical story of Shibboleth, where a tribe in conflict with another determines tribe affiliation by asking individuals to pronounce the word, 'Shibboleth.' In the story, failure to reproduce the word phonetically resulted in death. Here, we run five different versions of a 'Shibboleth' model: a first, simple version, which evaluates whether a composite variable of mimicry quality and detection quality is a superior predictor to the model's outcome than is cost of detection. The models thereafter evaluate variations on the simple model, incorporating group-level behaviours such as altruistic punishment. Our results suggest that group members' sensitivity to mimicry of the Shibboleth-signal is a better predictor of whether any signal of group identity goes into fixation in the overall population than is the cost of mimicry detection. Thus, the likelihood of being detected as a mimic may be more important than the costs imposed on mimics who are detected. This suggests that theoretical models in biology should place greater emphasis on the likelihood of detection, which does not explicitly entail costs, rather than on the costs to individuals who are detected. From a language learning perspective, the results suggest that admission to group membership through linguistic signals is powered by the ability to imitate and evade detection as an outsider by existing group members.

Timing in Conversation

Turn-taking in everyday conversation is fast, with median latencies in corpora of conversational speech often reported to be under 300 ms. This seems like magic, given that experimental research on speech planning has shown that speakers need much more time to plan and produce even the shortest of utterances. This paper reviews how language scientists have combined linguistic analyses of conversations and experimental work to understand the skill of swift turn-taking and proposes a tentative solution to the riddle of fast turn-taking.

EEG Correlates of Distractions and Hesitations in Human–Robot Interaction: A LabLinking Pilot Study

In this paper, we investigate the effect of distractions and hesitations as a scaffolding strategy. Recent research points to the potential beneficial effects of a speaker’s hesitations on the listeners’ comprehension of utterances, although results from studies on this issue indicate that humans do not make strategic use of them. The role of hesitations and their communicative function in human-human interaction is a much-discussed topic in current research. To better understand the underlying cognitive processes, we developed a human–robot interaction (HRI) setup that allows the measurement of the electroencephalogram (EEG) signals of a human participant while interacting with a robot. We thereby address the research question of whether we find effects on single-trial EEG based on the distraction and the corresponding robot’s hesitation scaffolding strategy. To carry out the experiments, we leverage our LabLinking method, which enables interdisciplinary joint research between remote labs. This study could not have been conducted without LabLinking, as the two involved labs needed to combine their individual expertise and equipment to achieve the goal together. The results of our study indicate that the EEG correlates in the distracted condition are different from the baseline condition without distractions. Furthermore, we could differentiate the EEG correlates of distraction with and without a hesitation scaffolding strategy. This proof-of-concept study shows that LabLinking makes it possible to conduct collaborative HRI studies in remote laboratories and lays the first foundation for more in-depth research into robotic scaffolding strategies.

Online Comprehension of Verbal Number Morphology in Children With Developmental Language Disorder: An Eye-Tracking Study

PURPOSE

Previous studies have raised the possibility of preserved language comprehension in children with developmental language disorder (DLD) in online tasks and within simple sentence structures. Consequently, we evaluated the capacity of children with DLD to comprehend verbal number agreement in simple sentence structures (i.e., verb-object-subject and verb-subject).

METHOD

Using an eye-tracking methodology, we conducted two psycholinguistic experiments with 96 Spanish- and Catalan-speaking participants. The sample was distributed into four groups: 24 children with DLD (age range: 4;6-12;6 [years;months]; average age = 7;8 [years;months]), 24 children with the same chronological age (4;6-12;2, 7;8), 24 children with the same linguistic level (4;6-9;4, 6;8), and 24 university students as language experts (18-30, 22;5).

RESULTS

The experimental data indicate that children with DLD can comprehend verbal number agreement at least under the present experimental conditions.

CONCLUSION

The empirical outcomes suggest that number morphology comprehension by children with DLD might be more typical than what it is generally considered to be.

A New Look at Young Children's Referential Informativeness

In this article, I review experimental evidence for the dependence of 2- to 5-year-olds' linguistic referential informativeness on cues to common ground (CG) and propose a process model. Cues to CG provide evidence for CG, that is, for the shared knowledge, beliefs, and attitudes of interlocutors. The presence of cues to CG (e.g., unimpeded listener line of regard or prior mention) is shown to be associated with less informative reference (e.g., pronouns). In contrast, the absence of cues to CG (e.g., impeded listener line of regard or new mention) is shown to be associated with more informative reference (e.g., nouns). Informativeness is sensitive to linguistic before nonlinguistic cues to CG (i.e., 2.0 vs. 2.5 years old, respectively). Reference is cast as a process of active inference, a formulation of Bayesian belief-guided control in biological systems. Child speakers are hierarchical generative models that, characteristically, expect sensory evidence for the evolved, prior Bayesian belief that interlocutor mental states are aligned (i.e., that CG exists). Referential control emerges as an embodied tool to gather evidence for this prior belief. Bottom-up cues to CG elicited by action drive updates to beliefs about CG. In turn, beliefs about CG guide efficient referential control.

Endogenous sources of interbrain synchrony in duetting pianists

When people interact with each other, their brains synchronize. However, it remains unclear whether interbrain synchrony (IBS) is functionally relevant for social interaction or stems from exposure of individual brains to identical sensorimotor information. To disentangle these views, the current dual-EEG study investigated amplitude-based IBS in pianists jointly performing duets containing a silent pause followed by a tempo change. First, we manipulated the similarity of the anticipated tempo change and measured IBS during the pause, hence, capturing the alignment of purely endogenous, temporal plans without sound or movement. Notably, right posterior gamma IBS was higher when partners planned similar tempi, it predicted whether partners' tempi matched after the pause, and it was modulated only in real, not in surrogate pairs. Second, we manipulated the familiarity with the partner's actions and measured IBS during joint performance with sound. Although sensorimotor information was similar across conditions, gamma IBS was higher when partners were unfamiliar with each other's part and had to attend more closely to the sound of the performance. These combined findings demonstrate that IBS is not merely an epiphenomenon of shared sensorimotor information but can also hinge on endogenous, cognitive processes crucial for behavioral synchrony and successful social interaction.

Timing of brain entrainment to the speech envelope during speaking, listening and self-listening

⁠This study investigates the dynamics of speech envelope tracking during speech production, listening and self-listening. We use a paradigm in which participants listen to natural speech (Listening), produce natural speech (Speech Production), and listen to the playback of their own speech (Self-Listening), all while their neural activity is recorded with EEG. After time-locking EEG data collection and auditory recording and playback, we used a Gaussian copula mutual information measure to estimate the relationship between information content in the EEG and auditory signals. In the 2-10 Hz frequency range, we identified different latencies for maximal speech envelope tracking during speech production and speech perception. Maximal speech tracking takes place approximately 110 ms after auditory presentation during perception and 25 ms before vocalisation during speech production. These results describe a specific timeline for speech tracking in speakers and listeners in line with the idea of a speech chain and hence, delays in communication.

The Seeds of the Noun–Verb Distinction in the Manual Modality: Improvisation and Interaction in the Emergence of Grammatical Categories

The noun–verb distinction has long been considered a fundamental property of human language, and has been found in some form even in the earliest stages of language emergence, including homesign and the early generations of emerging sign languages. We present two experimental studies that use silent gesture to investigate how noun–verb distinctions develop in the manual modality through two key processes: (i) improvising using novel signals by individuals, and (ii) using those signals in the interaction between communicators. We operationalise communicative interaction in two ways: a setting in which members of the dyad were in separate booths and were given a comprehension test after each stimulus vs. a more naturalistic face-to-face conversation without comprehension checks. There were few differences between the two conditions, highlighting the robustness of the paradigm. Our findings from both experiments reflect patterns found in naturally emerging sign languages. Some formal distinctions arise in the earliest stages of improvisation and do not require interaction to develop. However, the full range of formal distinctions between nouns and verbs found in naturally emerging language did not appear with either improvisation or interaction, suggesting that transmitting the language to a new generation of learners might be necessary for these properties to emerge.

Capturing the Temporal Dynamics of Language Style Matching in Groups and Teams

This article presents a dynamic conceptualization for the assessment of language style matching (LSM) over time. LSM is a team’s mutual adaption of function words like pronouns, articles, or prepositions. LSM is a nonconsciously but frequently occurring communication behavior allowing researchers unobtrusive insights into teams’ internal dynamics. Building on guidelines for the alignment of construct and measurement, a dynamic conceptualization and method for LSM are introduced. Simulated examples and interactions of N = 160 individuals in 26 teams indicate that dynamic LSM allows for a truer estimation of LSM than the hitherto used static method. Implications for future application are discussed.

Effects of verbal tasks on driving simulator performance

We report results from a driving simulator paradigm we developed to test the fine temporal effects of verbal tasks on simultaneous tracking performance. A total of 74 undergraduate students participated in two experiments in which they controlled a cursor using the steering wheel to track a moving target and where the dependent measure was overall deviation from target. Experiment 1 tested tracking performance during slow and fast target speeds under conditions involving either no verbal input or output, passive listening to spoken prompts via headphones, or responding to spoken prompts. Experiment 2 was similar except that participants read written prompts overlain on the simulator screen instead of listening to spoken prompts. Performance in both experiments was worse during fast speeds and worst overall during responding conditions. Most significantly, fine scale time-course analysis revealed deteriorating tracking performance as participants prepared and began speaking and steadily improving performance while speaking. Additionally, post-block survey data revealed that conversation recall was best in responding conditions, and perceived difficulty increased with task complexity. Our study is the first to track temporal changes in interference at high resolution during the first hundreds of milliseconds of verbal production and comprehension. Our results are consistent with load-based theories of multitasking performance and show that language production, and, to a lesser extent, language comprehension tap resources also used for tracking. More generally, our paradigm provides a useful tool for measuring dynamical changes in tracking performance during verbal tasks due to the rapidly changing resource requirements of language production and comprehension.

Tonal Emergence: An agent-based model of tonal coordination

Humans have a remarkable capacity for coordination. Our ability to interact and act jointly in groups is crucial to our success as a species. Joint Action (JA) research has often concerned itself with simplistic behaviors in highly constrained laboratory tasks. But there has been a growing interest in understanding complex coordination in more open-ended contexts. In this regard, collective music improvisation has emerged as a fascinating model domain for studying basic JA mechanisms in an unconstrained and highly sophisticated setting. A number of empirical studies have begun to elucidate coordination mechanisms underlying joint musical improvisation, but these findings have yet to be cached out in a working computational model. The present work fills this gap by presenting Tonal Emergence, an idealized agent-based model of improvised musical coordination. Tonal Emergence models the coordination of notes played by improvisers to generate harmony (i.e., tonality), by simulating agents that stochastically generate notes biased towards maximizing harmonic consonance given their partner's previous notes. The model replicates an interesting empirical result from a previous study of professional jazz pianists: feedback loops of mutual adaptation between interacting agents support the production of consonant harmony. The model is further explored to show how complex tonal dynamics, such as the production and dissolution of stable tonal centers, are supported by agents that are characterized by (i) a tendency to strive toward consonance, (ii) stochasticity, and (iii) a limited memory for previously played notes. Tonal Emergence thus provides a grounded computational model to simulate and probe the coordination mechanisms underpinning one of the more remarkable feats of human cognition: collective music improvisation.

Does it pay to imitate? No evidence for social gains from lexical imitation

According to an influential hypothesis, people imitate motor movements to foster social interactions. Could imitation of language serve a similar function? We investigated this question in two pre-registered experiments. In Experiment 1, participants were asked to alternate naming pictures and matching pictures to a name provided by a partner. Crucially, and unknown to participants, the partner was in fact a computer program which in one group produced the same names as previously used by the participant, and in the other group consistently produced different names. We found no difference in how the two groups evaluated the partner or the interaction and no difference in their willingness to cooperate with the partner. In Experiment 2, we made the task more similar to natural interactions by adding a stage in which a participant and the partner introduced themselves to each other and included a measure of the participant's autistic traits. Once again, we found no effects of being imitated. We discuss how these null results may inform imitation research.

Competition Reduces Response Times in Multiparty Conversation

Natural conversations are characterized by short transition times between turns. This holds in particular for multi-party conversations. The short turn transitions in everyday conversations contrast sharply with the much longer speech onset latencies observed in laboratory studies where speakers respond to spoken utterances. There are many factors that facilitate speech production in conversational compared to laboratory settings. Here we highlight one of them, the impact of competition for turns. In multi-party conversations, speakers often compete for turns. In quantitative corpus analyses of multi-party conversation, the fastest response determines the recorded turn transition time. In contrast, in dyadic conversations such competition for turns is much less likely to arise, and in laboratory experiments with individual participants it does not arise at all. Therefore, all responses tend to be recorded. Thus, competition for turns may reduce the recorded mean turn transition times in multi-party conversations for a simple statistical reason: slow responses are not included in the means. We report two studies illustrating this point. We first report the results of simulations showing how much the response times in a laboratory experiment would be reduced if, for each trial, instead of recording all responses, only the fastest responses of several participants responding independently on the trial were recorded. We then present results from a quantitative corpus analysis comparing turn transition times in dyadic and triadic conversations. There was no significant group size effect in question-response transition times, where the present speaker often selects the next one, thus reducing competition between speakers. But, as predicted, triads showed shorter turn transition times than dyads for the remaining turn transitions, where competition for the floor was more likely to arise. Together, these data show that turn transition times in conversation should be interpreted in the context of group size, turn transition type, and social setting.

Active Inference and Cooperative Communication: An Ecological Alternative to the Alignment View

We present and contrast two accounts of cooperative communication, both based on Active Inference, a framework that unifies biological and cognitive processes. The mental alignment account, defended in Vasil et al., takes the function of cooperative communication to be the alignment of the interlocutor's mental states, and cooperative communicative behavior to be driven by an evolutionarily selected adaptive prior belief favoring the selection of action policies that promote such an alignment. We argue that the mental alignment account should be rejected because it neglects the action-oriented nature of cooperative communication, which skews its view of the dynamics of communicative interaction. We introduce our own conception of cooperative communication, inspired by a more radical ecological interpretation of the active inference framework. Cooperative communication, on our ecological conception, serves to guide and constrain the dynamics of the cooperative interaction via the construction and restructuring of shared fields of affordances, in order to reach the local goals of the joint actions in which episodes of cooperative communication are embedded. We argue that our ecological conception provides a better theoretical standpoint to account for the action-oriented nature of cooperative communication in the active inference framework.

Hybrid Harmony: A Multi-Person Neurofeedback Application for Interpersonal Synchrony

Recent years have seen a dramatic increase in studies measuring brain activity, physiological responses, and/or movement data from multiple individuals during social interaction. For example, so-called "hyperscanning" research has demonstrated that brain activity may become synchronized across people as a function of a range of factors. Such findings not only underscore the potential of hyperscanning techniques to capture meaningful aspects of naturalistic interactions, but also raise the possibility that hyperscanning can be leveraged as a tool to help improve such naturalistic interactions. Building on our previous work showing that exposing dyads to real-time inter-brain synchrony neurofeedback may help boost their interpersonal connectedness, we describe the biofeedback application Hybrid Harmony, a Brain-Computer Interface (BCI) that supports the simultaneous recording of multiple neurophysiological datastreams and the real-time visualization and sonification of inter-subject synchrony. We report results from 236 dyads experiencing synchrony neurofeedback during naturalistic face-to-face interactions, and show that pairs' social closeness and affective personality traits can be reliably captured with the inter-brain synchrony neurofeedback protocol, which incorporates several different online inter-subject connectivity analyses that can be applied interchangeably. Hybrid Harmony can be used by researchers who wish to study the effects of synchrony biofeedback, and by biofeedback artists and serious game developers who wish to incorporate multiplayer situations into their practice.

Should I focus on self-language actions or should I follow others? Cross-language interference effects in voluntary and cued language switching

We examined whether and how language produced by others influences self-language processes. This study addressed this issue by looking at effects of comprehension on language switching in cued and voluntary switching contexts. During voluntary language switching, Chinese-English bilinguals were more likely to repeat the language they previously used themselves than to repeat the language produced by others. Furthermore, during both voluntary and cued language switching, bilinguals showed larger switch costs when switching between languages themselves than when switching after hearing another language. This suggests that cross-language interference may primarily stem from the self-language system rather than from language produced by others.

Constructing a protolanguage: reconstructing prehistoric languages in a usage-based construction grammar framework

Construction grammar is an approach to language that posits that units and structures in language can be exhaustively described as pairings between form and meaning. These pairings are called constructions and can have different degrees of abstraction, i.e. they span the entire range from very concrete (armadillo, avocado) to very abstract constructions such as the ditransitive construction (I gave her a book). This approach has been applied to a wide variety of different areas of research in linguistics, such as how new constructions emerge and change historically. It has also been applied to investigate the evolutionary emergence of modern fully fledged language, i.e. the question of how systems of constructions can arise out of prelinguistic communication. In this paper, we review the contribution of usage-based construction grammar approaches to language change and language evolution to the questions of (i) the structure and nature of prehistoric languages and (ii) how constructions in prehistoric languages emerged out of non-linguistic or protolinguistic communication. In particular, we discuss the possibilities of using constructions as the main unit of analysis both in reconstructing predecessors of existing languages (protolanguages) and in formulating theories of how a potential predecessor of human language in general (protolanguage) must have looked like. This article is part of the theme issue 'Reconstructing prehistoric languages'.

Team Physiological Dynamics: A Critical Review

OBJECTIVE

Review the use of physiological measurement in team settings and propose recommendations to improve the state of the science.

BACKGROUND

New sensor and analytical capabilities enable exploration of relationships between team members' physiological dynamics. We conducted a review of physiological measures used in research on teams to understand (1) how these measures are theoretically and operationally related to team constructs and (2) what types of validity evidence exist for physiological measurement in team settings.

METHOD

We identified 32 articles that investigated task-performing teams using physiological data. Articles were coded on several dimensions, including team characteristics. Study findings were categorized by relationships tested between team physiological dynamics (TPD) and team inputs, mediators/processes, outputs, or psychometric properties.

RESULTS

TPD researchers overwhelmingly measure single physiological systems. Although there is research linking TPD to inputs and outputs, the research on processes is underdeveloped.

CONCLUSION

We recommend several theoretical, methodological, and statistical themes to expand the growth of the TPD field.

APPLICATION

Physiological measures, once established as reliable indicators of team functioning, might be used to diagnose suboptimal team states and cue interventions to ameliorate these states.

Avoiding Conflict: When Speaker Coordination Does Not Require Conceptual Agreement

In this paper we discuss the socialization hypothesis-the idea that speakers of the same (linguistic) community should share similar concepts given that they are exposed to similar environments and operate in highly-coordinated social contexts-and challenge the fact that it is assumed to constitute a prerequisite to successful communication. We do so using distributional semantic models of meaning (DSMs) which create lexical representations via latent aggregation of co-occurrence information between words and contexts. We argue that DSMs constitute particularly adequate tools for exploring the socialization hypothesis given that 1) they provide full control over the notion of background environment, formally characterized as the training corpus from which distributional information is aggregated; and 2) their geometric structure allows for exploiting alignment-based similarity metrics to measure inter-subject alignment over an entire semantic space, rather than a set of limited entries. We propose to model coordination between two different DSMs trained on two distinct corpora as dimensionality selection over a dense matrix obtained via Singular Value Decomposition This approximates an ad-hoc coordination scenario between two speakers as the attempt to align their similarity ratings on a set of word pairs. Our results underline the specific way in which linguistic information is spread across singular vectors, and highlight the need to distinguish agreement from mere compatibility in alignment-based notions of conceptual similarity. Indeed, we show that compatibility emerges from idiosyncrasy so that the unique and distinctive aspects of speakers' background experiences can actually facilitate-rather than impede-coordination and communication between them. We conclude that the socialization hypothesis may constitute an unnecessary prerequisite to successful communication and that, all things considered, communication is probably best formalized as the cooperative act of avoiding conflict, rather than maximizing agreement.

Innovation and enculturation in child communication: a cross-sectional study

How can people achieve successful communication when using novel signs? Previous studies show that iconic signs (i.e. signs that directly resemble their referent) enhance communication success. In this paper, we test if enculturated signs (i.e. signs informed by interlocutors' shared culture) also enhance communication success. Children, who have spent less time in their linguistic community, have less cultural knowledge to inform their sign innovation. A natural prediction is that younger children's signs will be less enculturated, more diverse and less successful compared with older children and adults. We examined sign innovation in children aged between 6 and 12 years (N = 54) and adults (N = 18). Sign enculturation, diversity and iconicity were rated. As predicted, younger children innovated less enculturated and more diverse signs, and communicated less successfully than older children and adults. Sign enculturation and iconicity uniquely contributed to communication success. This is the first study to demonstrate that enculturated signs enhance communication.

Forgotten Little Words: How Backchannels and Particles May Facilitate Speech Planning in Conversation?

In everyday conversation, turns often follow each other immediately or overlap in time. It has been proposed that speakers achieve this tight temporal coordination between their turns by engaging in linguistic dual-tasking, i.e., by beginning to plan their utterance during the preceding turn. This raises the question of how speakers manage to co-ordinate speech planning and listening with each other. Experimental work addressing this issue has mostly concerned the capacity demands and interference arising when speakers retrieve some content words while listening to others. However, many contributions to conversations are not content words, but backchannels, such as “hm”. Backchannels do not provide much conceptual content and are therefore easy to plan and respond to. To estimate how much they might facilitate speech planning in conversation, we determined their frequency in a Dutch and a German corpus of conversational speech. We found that 19% of the contributions in the Dutch corpus, and 16% of contributions in the German corpus were backchannels. In addition, many turns began with fillers or particles, most often translation equivalents of “yes” or “no,” which are likewise easy to plan. We proposed that to generate comprehensive models of using language in conversation psycholinguists should study not only the generation and processing of content words, as is commonly done, but also consider backchannels, fillers, and particles.

Coordination and Consonance Between Interacting, Improvising Musicians

Joint action (JA) is ubiquitous in our cognitive lives. From basketball teams to teams of surgeons, humans often coordinate with one another to achieve some common goal. Idealized laboratory studies of group behavior have begun to elucidate basic JA mechanisms, but little is understood about how these mechanisms scale up in more sophisticated and open-ended JA that occurs in the wild. We address this gap by examining coordination in a paragon domain for creative joint expression: improvising jazz musicians. Coordination in jazz music subserves an aesthetic goal: the generation of a collective musical expression comprising coherent, highly nuanced musical structure (e.g., rhythm, harmony). In our study, dyads of professional jazz pianists improvised in a "coupled," mutually adaptive condition, and an "overdubbed" condition that precluded mutual adaptation, as occurs in common studio recording practices. Using a model of musical tonality, we quantify the flow of rhythmic and harmonic information between musicians as a function of interaction condition. Our analyses show that mutually adapting dyads achieve greater temporal alignment and produce more consonant harmonies. These musical signatures of coordination were preferred by independent improvisers and naive listeners, who gave higher quality ratings to coupled interactions despite being blind to condition. We present these results and discuss their implications for music technology and JA research more generally.

Generative models, linguistic communication and active inference

This paper presents a biologically plausible generative model and inference scheme that is capable of simulating communication between synthetic subjects who talk to each other. Building on active inference formulations of dyadic interactions, we simulate linguistic exchange to explore generative models that support dialogues. These models employ high-order interactions among abstract (discrete) states in deep (hierarchical) models. The sequential nature of language processing mandates generative models with a particular factorial structure-necessary to accommodate the rich combinatorics of language. We illustrate linguistic communication by simulating a synthetic subject who can play the 'Twenty Questions' game. In this game, synthetic subjects take the role of the questioner or answerer, using the same generative model. This simulation setup is used to illustrate some key architectural points and demonstrate that many behavioural and neurophysiological correlates of linguistic communication emerge under variational (marginal) message passing, given the right kind of generative model. For example, we show that theta-gamma coupling is an emergent property of belief updating, when listening to another.

Disrupted joint action accounts for reduced likability of socially anxious individuals

BACKGROUND AND OBJECTIVES

Motivated by their fear of disapproval, individuals with social anxiety continually monitor their own behaviour during social interactions hoping to prevent dreaded negative outcomes. Ironically, they do evoke less positive reactions from others. This study investigated whether lower engagement in the interpersonal process of joint action by socially anxious individuals leads them to attract less positive ratings by their conversation partners.

METHOD

High socially anxious (HSA; N = 36) and low socially anxious individuals (LSA; N = 36) had separate conversations with a naïve conversation partner (N = 36). Conversations were filmed and analysed for joint action using the conventional manual way and a more exploratory automated way. Conversation partners rated the quality of the interaction and the person they talked to.

RESULTS

Conversation partners rated HSA participants less positively than LSA participants. The conventional manual method showed less joint action in conversations with LSA participants and crucially, joint action served as mediator between social anxiety status and general impression, quality of interaction and desire to meet again. These results were not replicated with the automated method.

LIMITATIONS

The study used an analogue sample and future research should investigate whether these findings also apply to a clinical sample. Future studies should further explore the utility of automated techniques to measure joint action.

CONCLUSION

Reduced joint action may explain why socially anxious individuals tend to be perceived less positively by others. The findings emphasise the importance of taking an interpersonal and holistic approach to understanding aspects of social anxiety disorder.

Making oneself predictable in linguistic interactions

While language production is a highly demanding task, conversational partners are known to coordinate their turns with striking precision. Among the mechanisms that allow them to do so is listeners' ability to predict what the speaker will say, and thus to prepare their response in advance. But do speakers also play a role in facilitating coordination? We hypothesized that speakers contribute by using coordination smoothers - in particular by making their turns easier to predict. To test this, we asked participants to type definitions for common English words, either on their own (n = 26 individuals) or interacting with a partner (n = 18 pairs), and we measured the timing with which they produced the definitions. In a post-test, additional participants (n = 55) attempted to predict the final word of these definitions and rated them for quality. We found that interacting speakers initiated their turns with less variable delays than solo individuals. In contrast, our post-test measures suggested that jointly produced definitions were in fact of lower predictability and quality than those produced by individuals, but the analysis revealed these findings were likely confounded by task difficulty. We propose that the reduction in temporal variability observed for interacting speakers may facilitate prediction and thus act as a coordination smoother in linguistic interactions.

Real time comprehension of Spanish articles in children with developmental language disorder: Empirical evidence from eye movements

Function words and, more specifically, articles have been widely indicated as one of the main sources of difficulty for children with Developmental Language Disorder (DLD). The present study is the first to assess the online comprehension of Spanish articles in bilingual children with DLD. In an eye tracking experiment, we monitored participants' eye movements as they listened to Spanish articles embedded in structurally simple sentences. Ninety-six subjects from four different groups were evaluated: 24 children with DLD (average age 7;08), 24 children with the same chronological age (average age 7;08), 24 younger children matched for mean length utterance (average age 6;08), and 24 adults (average age 22;05). We calculated the proportion between the preference for the correct visual referent and a competitor object. Our results suggest that children with DLD are capable of timely comprehension of Spanish articles in real time and within simple sentence structures. However, we observed a strong effect of chronological age in the sample of interest; younger children with DLD are able to identify the correct referent, but this preference is weaker compared to the older children with DLD. We also observed local differences between the DLD group, and the other two children control groups, particularly when the chronological age group is introduced as a factor. These findings suggest a developmental trajectory that is different in the DLD group relative to children with typical language development. Notwithstanding, in spite of the article production difficulties previously reported, children with DLD in the present sample appear to be able to comprehend Spanish articles in the current experimental conditions.

Finding Structure in Time: Visualizing and Analyzing Behavioral Time Series

The temporal structure of behavior contains a rich source of information about its dynamic organization, origins, and development. Today, advances in sensing and data storage allow researchers to collect multiple dimensions of behavioral data at a fine temporal scale both in and out of the laboratory, leading to the curation of massive multimodal corpora of behavior. However, along with these new opportunities come new challenges. Theories are often underspecified as to the exact nature of these unfolding interactions, and psychologists have limited ready-to-use methods and training for quantifying structures and patterns in behavioral time series. In this paper, we will introduce four techniques to interpret and analyze high-density multi-modal behavior data, namely, to: (1) visualize the raw time series, (2) describe the overall distributional structure of temporal events (Burstiness calculation), (3) characterize the non-linear dynamics over multiple timescales with Chromatic and Anisotropic Cross-Recurrence Quantification Analysis (CRQA), (4) and quantify the directional relations among a set of interdependent multimodal behavioral variables with Granger Causality. Each technique is introduced in a module with conceptual background, sample data drawn from empirical studies and ready-to-use Matlab scripts. The code modules showcase each technique's application with detailed documentation to allow more advanced users to adapt them to their own datasets. Additionally, to make our modules more accessible to beginner programmers, we provide a "Programming Basics" module that introduces common functions for working with behavioral timeseries data in Matlab. Together, the materials provide a practical introduction to a range of analyses that psychologists can use to discover temporal structure in high-density behavioral data.

Interactive Alignment and Lexical Triggering of Code-Switching in Bilingual Dialogue

When bilingual speakers use two languages in the same utterance, this is called code-switching. Previous research indicates that bilinguals’ likelihood to code-switch is enhanced when the utterance to be produced (1) contains a word with a similar form across languages (lexical triggering) and (2) is preceded by a code-switched utterance, for example from a dialogue partner (interactive alignment/priming of code-switching). Both factors have mostly been tested on corpus data and have not yet been studied in combination. In two experiments, we therefore investigated the combined effects of interactive alignment and lexical triggering on code-switching. In Experiment 1, Dutch-English bilinguals described pictures to each other in a dialogue game where a confederate’s code-switching was manipulated. The participants were free to use either Dutch, English, or a combination of Dutch and English in describing the pictures, so they could voluntarily code-switch or not. The pictures contained a cognate [e.g., roos (rose)], a false friend [e.g., rok (skirt, false friend with rock)], or a control word [e.g., jas (coat)]. Participants code-switched more often when the confederate had just code-switched (indicating interactive alignment). They also code-switched more often when cognates were involved, but only when the confederate had just code-switched. This indicates that lexical triggering is driven by interactive alignment. False friends did not enhance the likelihood of code-switching. Experiment 2 used a similar dialogue game with participants from the same population but focused specifically on how to account for interactive alignment of code-switching. Rather than aligning on their dialogue partner’s pragmatic act of code-switching, bilinguals aligned on the language activation from the utterance produced by their dialogue partner. All in all, the results show how co-activation of languages at multiple levels of processing together influence bilinguals’ tendency to code-switch. The findings call for a perspective on bilingual language production in which cross-speaker and cross-language processes are combined.

Is Language Production Planning Emergent From Action Planning? A Preliminary Investigation

The nature of syntactic planning for language production may reflect language-specific processes, but an alternative is that syntactic planning is an example of more domain-general action planning processes. If so, language and non-linguistic action planning should have identifiable commonalities, consistent with an underlying shared system. Action and language research have had little contact, however, and such comparisons are therefore lacking. Here, we address this gap by taking advantage of a striking similarity between two phenomena in language and action production. One is known as syntactic priming-the tendency to re-use a recently produced sentence structure-and the second is hysteresis-the tendency to re-use a previously executed abstract action plan, such as a limb movement. We examined syntactic priming/hysteresis in parallel language and action tasks intermixed in a single experimental session. Our goals were to establish the feasibility of investigating language and action planning within the same participants and to inform debates on the language-specific vs. domain-general nature of planning systems. In both action and language tasks, target trials afforded two alternative orders of subcomponents in the participant's response: in the language task, a picture could be described with two different word orders, and in the action task, locations on a touch screen could be touched in two different orders. Prime trials preceding the target trial promoted one of two plans in the respective domain. Manipulations yielded higher rates of primed behavior in both tasks. In an exploratory cross-domain analysis, there was some evidence for stronger priming effects in some combinations of action and language priming conditions than others. These results establish a method for investigating the degree to which language planning is part of a domain-general action planning system.

Prosocial effects of coordination - What, how and why?

A wealth of research in recent decades has investigated the effects of various forms of coordination upon prosocial attitudes and behavior. To structure and constrain this research, we provide a framework within which to distinguish and interrelate different hypotheses about the psychological mechanisms underpinning various prosocial effects of various forms of coordination. To this end, we introduce a set of definitions and distinctions that can be used to tease apart various forms of prosociality and coordination. We then identify a range of psychological mechanisms that may underpin the effects of coordination upon prosociality. We show that different hypotheses about the underlying psychological mechanisms motivate different predictions about the effects of various forms of coordination in different circumstances.

Tilting Together: An Information-Theoretic Characterization of Behavioral Roles in Rhythmic Dyadic Interaction

Every joint collaborative physical activity performed by a group of people, e.g., carrying a table, typically leads to the emergence of spatiotemporal coordination of individual motor behavior. Such interpersonal coordination can arise solely based on the observation of the partners' and/or the object's movements, without the presence of verbal communication. In this paper, we investigate how the social coupling between two individuals in a collaborative task translates into measured objective and subjective performance indicators recorded in two different studies. We analyse the trends in the dyadic interrelationship based on the information-theoretic measure of transfer entropy and identify emerging leader-follower roles. In our experimental paradigm, the actions of the pair of subjects are continuously and seamlessly fused, resulting in a joint control of an object simulated on a tablet computer. Subjects need to synchronize their movements with a 90° phase difference in order to keep the object (a ball) rotating precisely on a predefined circular or elliptic trajectory on a tablet device. Results demonstrate how the identification of causal dependencies in this social interaction task could reveal specific trends in human behavior and provide insights into the emergence of social sensorimotor contingencies.

Role of joint language control during cross-language communication: evidence from cross-frequency coupling

How do bilingual interlocutors inhibit interference from the non-target language to achieve brain-to-brain information exchange in a task to simulate a bilingual speaker-listener interaction. In the current study, two electroencephalogram devices were employed to record pairs of participants' performances in a joint language switching task. Twenty-eight (14 pairs) unbalanced Chinese-English bilinguals (L1 Chinese) were instructed to name pictures in the appropriate language according to the cue. The phase-amplitude coupling analysis was employed to reveal the large-scale brain network responsible for joint language control between interlocutors. We found that (1) speakers and listeners coordinately suppressed cross-language interference through cross-frequency coupling, as shown in the increased delta/theta phase-amplitude and delta/alpha phase-amplitude coupling when switching to L2 than switching to L1; (2) speakers and listeners were both able to simultaneously inhibit cross-person item-level interference which was demonstrated by stronger cross-frequency coupling in the cross person condition compared to the within person condition. These results indicate that current bilingual models (e.g., the inhibitory control model) should incorporate mechanisms that address inhibiting interference sourced in both language and person (i.e., cross-language and cross-person item-level interference) synchronously through joint language control in dynamic cross-language communication.

Musical Training for Auditory Rehabilitation in Hearing Loss

Despite the overall success of cochlear implantation, language outcomes remain suboptimal and subject to large inter-individual variability. Early auditory rehabilitation techniques have mostly focused on low-level sensory abilities. However, a new body of literature suggests that cognitive operations are critical for auditory perception remediation. We argue in this paper that musical training is a particularly appealing candidate for such therapies, as it involves highly relevant cognitive abilities, such as temporal predictions, hierarchical processing, and auditory-motor interactions. We review recent studies demonstrating that music can enhance both language perception and production at multiple levels, from syllable processing to turn-taking in natural conversation.

From speech and talkers to the social world: The neural processing of human spoken language

Human speech perception is a paradigm example of the complexity of human linguistic processing; however, it is also the dominant way of expressing vocal identity and is critically important for social interactions. Here, I review the ways that the speech, the talker, and the social nature of speech interact and how this may be computed in the human brain, using models and approaches from nonhuman primate studies. I explore the extent to which domain-general approaches may be able to account for some of these neural findings. Finally, I address the importance of extending these findings into a better understanding of the social use of speech in conversations.

Interaction Promotes the Adaptation of Referential Conventions to the Communicative Context

Coordination between speakers in dialogue requires balancing repetition and change, the old and the new. Interlocutors tend to reuse established forms, relying on communicative precedents. Yet linguistic interaction also necessitates adaptation to changing contexts or dynamic tasks, which might favor abandoning existing precedents in favor of better communicative alternatives. We explored this tension using a maze game task in which individual participants and interacting pairs had to describe figures and their positions in one of two possible maze types: a regular maze, in which the grid-like structure of the maze is highlighted, and an irregular maze, in which specific parts of the maze are salient. Participants repeated this task several times. Both individuals and interacting pairs were affected by the different maze layouts, initially using more idiosyncratic description schemes for irregular mazes and more systematic schemes for regular mazes. Interacting pairs, but not individuals, abandoned their unsystematic initial descriptions in favor of a more systematic approach, which was better adapted for repeated interaction. Our results show communicative conventions are initially shaped by context, but interaction opens up the possibility for change if better alternatives are available.

Conversation in Aphasia Across Communication Partners: Exploring Stability of Microlinguistic Measures and Communicative Success

Purpose The aim of this study was to determine if people with aphasia demonstrate differences in microlinguistic skills and communicative success in unstructured, nontherapeutic conversations with a home communication partner (Home-P) as compared to a speech-language pathologist communication partner (SLP-P). Method Eight persons with aphasia participated in 2 unstructured, nontherapeutic 15-minute conversations, 1 each with an unfamiliar SLP-P and a Home-P. Utterance-level analysis evaluated communicative success. Two narrow measures of lexical relevance and sentence frame were used to evaluate independent clauses. Two broad lexical and morphosyntactic measures were used to evaluate elliptical and dependent clauses and to evaluate independent clauses for errors beyond lexical relevance and sentence frame (such as phonological and morphosyntactic errors). Utterances were further evaluated for presence of behaviors indicating lexical retrieval difficulty (pauses, repetitions, and false starts) and for referential cohesion. Results No statistical differences occurred for communicative success or for any of the microlinguistic measures between the SLP-P and Home-P conversation conditions. Four measures (2 of lexical retrieval and 1 each of communicative success and grammaticality) showed high correlations across the 2 conversation samples. Individuals showed variation of no more than 10 percentage points between the 2 conversation conditions for 46 of 56 data points. Variation greater than 10 percentage points tended to occur for the measure of referential cohesion and primarily for 1 participant. Conclusions Preliminary findings suggest that these microlinguistic measures and communicative success have potential for reliable comparison across Home-P and SLP-P conversations, with the possible exception of referential cohesion. However, further research is needed with a larger, more diverse sample. These findings suggest future assessment and treatment implications for clinical and research needs. Supplemental Material https://doi.org/10.23641/asha.7616312.

The Conference of the Birds: An Old Artistic Concept Making Sense in Modern Sciences

In this article, we will discuss scientific aspects of an old Persian story, Simorgh, in the book of The Conference of the Birds. The story is fulfilled with artistic and philosophical metaphors that make sense in two hot topics of the contemporary modern sciences i.e. cognitive science and complexity science. The poet addresses some humanity’s bygone concerns and fundamental questions about self, the quality that shapes a person’s uniqueness, and essential existence. The sophisticated language used in the poem contains allusions, symbols, and implications that are interpreted in five main topics. We think that the story deserves to be the touchstone for questions on the nature of the mind, including the profound question of humanity’s search for self and meaning of life.