Quantifying time-varying coordination of multimodal speech signals using correlation map analysis

multiSyncPy: A Python package for assessing multivariate coordination dynamics

In order to support the burgeoning field of research into intra- and interpersonal synchrony, we present an open-source software package: multiSyncPy. Multivariate synchrony goes beyond the bivariate case and can be useful for quantifying how groups, teams, and families coordinate their behaviors, or estimating the degree to which multiple modalities from an individual become synchronized. Our package includes state-of-the-art multivariate methods including symbolic entropy, multidimensional recurrence quantification analysis, coherence (with an additional sum-normalized modification), the cluster-phase 'Rho' metric, and a statistical test based on the Kuramoto order parameter. We also include functions for two surrogation techniques to compare the observed coordination dynamics with chance levels and a windowing function to examine time-varying coordination for most of the measures. Taken together, our collation and presentation of these methods make the study of interpersonal synchronization and coordination dynamics applicable to larger, more complex and often more ecologically valid study designs. In this work, we summarize the relevant theoretical background and present illustrative practical examples, lessons learned, as well as guidance for the usage of our package - using synthetic as well as empirical data. Furthermore, we provide a discussion of our work and software and outline interesting further directions and perspectives. multiSyncPy is freely available under the LGPL license at: https://github.com/cslab-hub/multiSyncPy , and also available at the Python package index.

Modulation transfer functions for audiovisual speech

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

Speech Kinematics and Coordination Measured With an MEG-Compatible Speech Tracking System

Articulography and functional neuroimaging are two major tools for studying the neurobiology of speech production. Until recently, however, it has generally not been possible to use both in the same experimental setup because of technical incompatibilities between the two methodologies. Here we describe results from a novel articulography system dubbed Magneto-articulography for the Assessment of Speech Kinematics (MASK), which we used to derive kinematic profiles of oro-facial movements during speech. MASK was used to characterize speech kinematics in two healthy adults, and the results were compared to measurements from a separate participant with a conventional Electromagnetic Articulography (EMA) system. Analyses targeted the gestural landmarks of reiterated utterances /ipa/, /api/ and /pataka/. The results demonstrate that MASK reliably characterizes key kinematic and movement coordination parameters of speech motor control. Since these parameters are intrinsically registered in time with concurrent magnetoencephalographic (MEG) measurements of neuromotor brain activity, this methodology paves the way for innovative cross-disciplinary studies of the neuromotor control of human speech production, speech development, and speech motor disorders.

Intrapersonal Behavioral Coordination and Expressive Accuracy During First Impressions

What factors influence how accurately we express our personalities? Here, we investigated the role of targets’ nonverbal expressivity or the intrapersonal coordination between head and body movements. To do so, using a novel movement quantification method, we examined whether variability in a person’s behavioral coordination was related to how accurately their personality was perceived by naive observers. Targets who exhibited greater variability in intrapersonal behavior coordination, indicating more expressive behavior, were perceived more accurately on high observability personality items, such as how energetic and helpful they are. Moreover, these associations held controlling for other indicators of overall movement, self- and perceiver-rated extroversion, as well as how engaging and likable targets were perceived to be. This provides preliminary evidence that variability in intrapersonal behavioral coordination may be a unique behavioral indicator of expressive accuracy, although further research that replicates these findings and examines the causal associations is needed.

The Role of Temporal Modulation in Sensorimotor Interaction

How do we align the distinct neural patterns associated with the articulation and the acoustics of the same utterance in order to guide behaviors that demand sensorimotor interaction, such as vocal learning and the use of feedback during speech production? One hypothesis is that while the representations are distinct, their patterns of change over time (temporal modulation) are systematically related. This hypothesis is pursued in the exploratory study described here, using paired articulatory and acoustic data from the X-ray microbeam corpus. The results show that modulation in both articulatory movement and in the changing acoustics has the form of a pulse-like structure related to syllable structure. The pulses are aligned with each other in time, and the modulation functions are robustly correlated. These results encourage further investigation and testing of the hypothesis.

Kinematic Analysis of Pianists' Expressive Performances of Romantic Excerpts: Applications for Enhanced Pedagogical Approaches

Established pedagogical theories for classical piano usually do not consider the essential relationship between the musical structure, whole body movements, and expression. Research focusing on musicians' expression has shown that body movements reflect the performer's understanding of the musical structure. However, most studies to date focus on the performance of a single piece at a time, leaving unanswered the question on how structural parameters of pieces with varied technical difficulties influence pianists' movements. In this study, 10 pianists performed three contrasting Romantic excerpts in terms of technical level and character, while motion data was collected with a passive infrared motion capture system. We observed how pianists modulate their performances for each of the three pieces and measured the absolute difference in percentage of duration and quantity of motion (QoM) between four expressive conditions (normal, deadpan, exaggerated, immobile). We analyzed common patterns within the time-series of position data to investigate whether pianists embody musical structure in similar ways. A survey was filled in by pianists to understand how they conceive the relationship between body movements and musical structure. Results show that the variation in duration between the exaggerated and deadpan conditions was significant in one measure for one of the excerpts, and that tempo was less affected by the QoM used than by the level of expression. By applying PCA on the pianists' position data, we found that the head QoM is an important parameter for communicating different expressions and structural features. Significant variations in head QoM were found in the immobile and deadpan conditions if compared to the normal condition, only in specific regions of the score. Recurrent head movements occurred along with certain structural parameters for two of the excerpts only. Altogether, these results indicate that the analysis of pianists' body movements and expressive intentions should be carried out in relation to the specific musical context, being dependent on the technical level of the pieces and the repertoire. These results, combined with piano teaching methods, may lead to the development of new approaches in instrumental lessons to help students make independent choices regarding body movements and expression.

Quantitative Analysis of Multimodal Speech Data

This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.

Problem-Solving Phase Transitions During Team Collaboration

Multiple theories of problem-solving hypothesize that there are distinct qualitative phases exhibited during effective problem-solving. However, limited research has attempted to identify when transitions between phases occur. We integrate theory on collaborative problem-solving (CPS) with dynamical systems theory suggesting that when a system is undergoing a phase transition it should exhibit a peak in entropy and that entropy levels should also relate to team performance. Communications from 40 teams that collaborated on a complex problem were coded for occurrence of problem-solving processes. We applied a sliding window entropy technique to each team's communications and specified criteria for (a) identifying data points that qualify as peaks and (b) determining which peaks were robust. We used multilevel modeling, and provide a qualitative example, to evaluate whether phases exhibit distinct distributions of communication processes. We also tested whether there was a relationship between entropy values at transition points and CPS performance. We found that a proportion of entropy peaks was robust and that the relative occurrence of communication codes varied significantly across phases. Peaks in entropy thus corresponded to qualitative shifts in teams' CPS communications, providing empirical evidence that teams exhibit phase transitions during CPS. Also, lower average levels of entropy at the phase transition points predicted better CPS performance. We specify future directions to improve understanding of phase transitions during CPS, and collaborative cognition, more broadly.

Relating movement recurrence and expressive timing patterns in music performances

In this study the movement patterns of ten expert musicians are quantitatively related to expressive timing patterns and the music structure during performances. The hypothesis is that ancillary gestures recurrently employed are closely related to expressive intentions, and that the expressive content imposed in key musical passages is thus reflected in the patterns of gestural recurrence. A movement and an audio analysis of 30 clarinet performances of a Brahms' excerpt are compared. Results show direct correlations between the recurrence pattern of clarinetists' ancillary movements and expressive bar duration manipulations employed by them, associated with melodic phrasing and harmonic transitions.

"Diagnosis by behavioral observation" home-videosomnography - a rigorous ethnographic approach to sleep of children with neurodevelopmental conditions

INTRODUCTION

Advanced video technology is available for sleep-laboratories. However, low-cost equipment for screening in the home setting has not been identified and tested, nor has a methodology for analysis of video recordings been suggested.

METHODS

We investigated different combinations of hardware/software for home-videosomnography (HVS) and established a process for qualitative and quantitative analysis of HVS-recordings. A case vignette (HVS analysis for a 5.5-year-old girl with major insomnia and several co-morbidities) demonstrates how methodological considerations were addressed and how HVS added value to clinical assessment.

RESULTS

We suggest an "ideal set of hardware/software" that is reliable, affordable (∼$500) and portable (=2.8 kg) to conduct non-invasive HVS, which allows time-lapse analyses. The equipment consists of a net-book, a camera with infrared optics, and a video capture device. (1) We present an HVS-analysis protocol consisting of three steps of analysis at varying replay speeds: (a) basic overview and classification at 16× normal speed; (b) second viewing and detailed descriptions at 4-8× normal speed, and (c) viewing, listening, and in-depth descriptions at real-time speed. (2) We also present a custom software program that facilitates video analysis and note-taking (Annotator(©)), and Optical Flow software that automatically quantifies movement for internal quality control of the HVS-recording. The case vignette demonstrates how the HVS-recordings revealed the dimension of insomnia caused by restless legs syndrome, and illustrated the cascade of symptoms, challenging behaviors, and resulting medications.

CONCLUSION

The strategy of using HVS, although requiring validation and reliability testing, opens the floor for a new "observational sleep medicine," which has been useful in describing discomfort-related behavioral movement patterns in patients with communication difficulties presenting with challenging/disruptive sleep/wake behaviors.

Embodiment of intersubjective time: relational dynamics as attractors in the temporal coordination of interpersonal behaviors and experiences

This paper addresses the issue of “being together,” and more specifically the issue of “being together in time.” We provide with an integrative framework that is inspired by phenomenology, the enactive approach and dynamical systems theories. To do so, we first define embodiment as a living and lived phenomenon that emerges from agent-world coupling. We then show that embodiment is essentially dynamical and therefore we describe experiential, behavioral and brain dynamics. Both lived temporality and the temporality of the living appear to be complex, multiscale phenomena. Next we discuss embodied dynamics in the context of interpersonal interactions, and briefly review the empirical literature on between-persons temporal coordination. Overall, we propose that being together in time emerges from the relational dynamics of embodied interactions and their flexible co-regulation.

Cross-recurrence quantification analysis of categorical and continuous time series: an R package

This paper describes the R package crqa to perform cross-recurrence quantification analysis of two time series of either a categorical or continuous nature. Streams of behavioral information, from eye movements to linguistic elements, unfold over time. When two people interact, such as in conversation, they often adapt to each other, leading these behavioral levels to exhibit recurrent states. In dialog, for example, interlocutors adapt to each other by exchanging interactive cues: smiles, nods, gestures, choice of words, and so on. In order for us to capture closely the goings-on of dynamic interaction, and uncover the extent of coupling between two individuals, we need to quantify how much recurrence is taking place at these levels. Methods available in crqa would allow researchers in cognitive science to pose such questions as how much are two people recurrent at some level of analysis, what is the characteristic lag time for one person to maximally match another, or whether one person is leading another. First, we set the theoretical ground to understand the difference between “correlation” and “co-visitation” when comparing two time series, using an aggregative or cross-recurrence approach. Then, we describe more formally the principles of cross-recurrence, and show with the current package how to carry out analyses applying them. We end the paper by comparing computational efficiency, and results’ consistency, of crqa R package, with the benchmark MATLAB toolbox crptoolbox (Marwan, 2013). We show perfect comparability between the two libraries on both levels.

Application of concepts from cross-recurrence analysis in speech production: an overview and comparison with other nonlinear methods

PURPOSE

The aim of this article was to introduce an important tool, cross-recurrence analysis, to speech production applications by showing how it can be adapted to evaluate the similarity of multivariate patterns of articulatory motion. The method differs from classical applications of cross-recurrence analysis because no phase space reconstruction is conducted, and a cleaning algorithm removes the artifacts from the recurrence plot. The main features of the proposed approach are robustness to nonstationarity and efficient separation of amplitude variability from temporal variability.

METHOD

The authors tested these claims by applying their method to synthetic stimuli whose variability had been carefully controlled. The proposed method was also demonstrated in a practical application: It was used to investigate the role of biomechanical constraints in articulatory reorganization as a consequence of speeded repetition of CVCV utterances containing a labial and a coronal consonant.

RESULTS

Overall, the proposed approach provided more reliable results than other methods, particularly in the presence of high variability.

CONCLUSION

The proposed method is a useful and appropriate tool for quantifying similarity and dissimilarity in patterns of speech articulator movement, especially in such research areas as speech errors and pathologies, where unpredictable divergent behavior is expected.

Dynamic structure of joint-action stimulus-response activity

The mere presence of a co-actor can influence an individual’s response behavior. For instance, a social Simon effect has been observed when two individuals perform a Go/No-Go response to one of two stimuli in the presence of each other, but not when they perform the same task alone. Such effects are argued to provide evidence that individuals co-represent the task goals and the to-be-performed actions of a co-actor. Motivated by the complex-systems approach, the present study was designed to investigate an alternative hypothesis — that such joint-action effects are due to a dynamical (time-evolving) interpersonal coupling that operates to perturb the behavior of socially situated actors. To investigate this possibility, participants performed a standard Go/No-Go Simon task in joint and individual conditions. The dynamic structure of recorded reaction times was examined using fractal statistics and instantaneous cross-correlation. Consistent with our hypothesis that participants responding in a shared space would become behaviorally coupled, the analyses revealed that reaction times in the joint condition displayed decreased fractal structure (indicative of interpersonal perturbation processes modulating ongoing participant behavior) compared to the individual condition, and were more correlated across a range of time-scales compared to the reaction times of pseudo-pair controls. Collectively, the findings imply that dynamic processes might underlie social stimulus-response compatibility effects and shape joint cognitive processes in general.