Articulatory constraints on spontaneous entrainment between speech and manual gesture

Gesture-speech synchrony in schizophrenia: A pilot study using a kinematic-acoustic analysis

Severe impairment of social functioning is the core feature of schizophrenia that persists despite treatment, and contributes to chronic functional disability. Abnormal non-verbal behaviors have been reported during interpersonal interactions but the temporal coordination of co-speech gestures with language abilities have been poorly studied to date in this pathology. Using the dynamical systems framework, the goal of the current study was to investigate whether gestures and speech synchrony is impaired in schizophrenia, exploring a new approach to report communicational skill disorders. Performing the first continuous kinematic-acoustic analysis in individuals with schizophrenia, we examined gesture-speech synchrony in solo spontaneous speech and in sensorimotor synchronization task. The experimental group consisted of twenty-eight participants with a diagnosis of schizophrenia and the control group consisted of twenty-four healthy participants age-gender-education matched. The results showed that spontaneous gesture-speech synchrony was preserved while intentional finger tapping-speech synchrony was impaired. In sensorimotor synchronization task, the schizophrenia group displayed greater asynchronies between finger tapping and syllable uttering and lower stability of coordination patterns. These findings suggest a specific deficit in time delay of information circulation and processing, especially in explicit functions. Thus, investigating intrapersonal coordination in schizophrenia may constitute a promising window into brain/behavior dynamic relationship.

Gesture-vocal coupling in Karnatak music performance: A neuro-bodily distributed aesthetic entanglement

In many musical styles, vocalists manually gesture while they sing. Coupling between gesture kinematics and vocalization has been examined in speech contexts, but it is an open question how these couple in music making. We examine this in a corpus of South Indian, Karnatak vocal music that includes motion-capture data. Through peak magnitude analysis (linear mixed regression) and continuous time-series analyses (generalized additive modeling), we assessed whether vocal trajectories around peaks in vertical velocity, speed, or acceleration were coupling with changes in vocal acoustics (namely, F0 and amplitude). Kinematic coupling was stronger for F0 change versus amplitude, pointing to F0's musical significance. Acceleration was the most predictive for F0 change and had the most reliable magnitude coupling, showing a one-third power relation. That acceleration, rather than other kinematics, is maximally predictive for vocalization is interesting because acceleration entails force transfers onto the body. As a theoretical contribution, we argue that gesturing in musical contexts should be understood in relation to the physical connections between gesturing and vocal production that are brought into harmony with the vocalists' (enculturated) performance goals. Gesture-vocal coupling should, therefore, be viewed as a neuro-bodily distributed aesthetic entanglement.

The impact of pitch on tempo-spatial accuracy and precision in intercepting a virtually moving ball

In two experiments, horizontal and vertical orientated sounds moved in parabolas. Participants had to touch a screen to indicate where and when a virtual moving ball would cross a visible line. We predicted that due to the sensitivity of the auditory system to temporal information, manipulations of pitch should affect temporal errors more than spatial errors. Stimuli were sound sources at five different pitches moving along a parabola produced through loudspeakers mounted around a touch screen. Results showed pitch effects on spatial constant and spatial variable errors when the parabola was horizontally oriented (Exp. 1), and on temporal constant errors in vertically oriented parabolas (Exp. 2). We conclude that temporal and spatial precision in interception tasks were affected differently by pitch manipulations and require consideration in future studies when assessing the impact of auditory information on catching virtually moving balls.

Gesture-speech physics in fluent speech and rhythmic upper limb movements

It is commonly understood that hand gesture and speech coordination in humans is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech vocalization has been studied in steady-state vocalization and monosyllabic utterances, where forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (n = 37), we extend this previous line of work to show that gesture-speech physics also impacts fluent speech. Compared with nonmovement, participants who are producing fluent self-formulated speech while rhythmically moving their limbs demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lower-impulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.

The quantification of gesture-speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and timescales. For example, gesture’s kinematic peaks (e.g., maximum speed) are coupled with prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high-resolution measurement of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies, given that field’s classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., as compared to research on instrumental action). Often kinematic peaks in gesture are measured by eye, where a “moment of maximum effort” is determined by several raters. In the present article, we provide a tutorial on more efficient methods to quantify the temporal properties of gesture kinematics, in which we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that the videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the present article incites gesture researchers to embark on the widespread objective study of gesture kinematics and their relation to speech.

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

Expressive moments in communicative hand gestures often align with emphatic stress in speech. It has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impulses on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) repeatedly uttered consonant-vowel (/pa/) mono-syllables while moving in particular phase relations with speech, or not moving the upper limbs. This study shows that respiration-related activity is affected by (especially high-impulse) gesturing when vocalizations occur near peaks in physical impulse. This study further shows that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, tight relations between respiration-related activity and vocalization were observed, even in the absence of movement, but even more so when upper-limb movement is present. The current findings expand a developing line of research showing that speech is modulated by functional biomechanical linkages between hand gestures and the respiratory system. This identification of gesture-speech biomechanics promises to provide an alternative phylogenetic, ontogenetic, and mechanistic explanatory route of why communicative upper limb movements co-occur with speech in humans.

Entrainment and Modulation of Gesture-Speech Synchrony Under Delayed Auditory Feedback

Gesture–speech synchrony re‐stabilizes when hand movement or speech is disrupted by a delayed feedback manipulation, suggesting strong bidirectional coupling between gesture and speech. Yet it has also been argued from case studies in perceptual–motor pathology that hand gestures are a special kind of action that does not require closed‐loop re‐afferent feedback to maintain synchrony with speech. In the current pre‐registered within‐subject study, we used motion tracking to conceptually replicate McNeill's (1992) classic study on gesture–speech synchrony under normal and 150 ms delayed auditory feedback of speech conditions (NO DAF vs. DAF). Consistent with, and extending McNeill's original results, we obtain evidence that (a) gesture‐speech synchrony is more stable under DAF versus NO DAF (i.e., increased coupling effect), (b) that gesture and speech variably entrain to the external auditory delay as indicated by a consistent shift in gesture‐speech synchrony offsets (i.e., entrainment effect), and (c) that the coupling effect and the entrainment effect are co‐dependent. We suggest, therefore, that gesture–speech synchrony provides a way for the cognitive system to stabilize rhythmic activity under interfering conditions.

A flexible and accurate method to estimate the mode and stability of spontaneous coordinated behaviors: The index-of-stability (IS) analysis

Patterns of coordination result from the interaction between (at least) two oscillatory components. This interaction is typically understood by means of two variables: the mode that expresses the shape of the interaction, and the stability that is the robustness of the interaction in this mode. A potent method of investigating coordinated behaviors is to examine the extent to which patterns of coordination arise spontaneously. However, a prominent issue faced by researchers is that, to date, no standard methods exist to fairly assess the stability of spontaneous coordination. In the present study, we introduce a new method called the index-of-stability (IS) analysis. We developed this method from the phase-coupling (PC) analysis that has been traditionally used for examining locomotion-respiration coordinated systems. We compared the extents to which both methods estimate the stability of simulated coordinated behaviors. Computer-generated time series were used to simulate the coordination of two rhythmic components according to a selected mode m:n and a selected degree of stability. The IS analysis was superior to the PC analysis in estimating the stability of spontaneous coordinated behaviors, in three ways: First, the estimation of stability itself was found to be more accurate and more reliable with the IS analysis. Second, the IS analysis is not constrained by the limitations of the PC analysis. Third and last, the IS analysis offers more flexibility, and so can be adapted according to the user's needs.

Influence of pacer continuity on continuous and discontinuous visuo-motor synchronisation

Previous research has reported that synchronising movements with an external pacer, known as sensorimotor synchronisation (SMS), is more stable when the movements are discrete/discontinuous rather than continuous. A standard explanation considers that more efficient mechanisms are involved for regulating synchronisation when producing discontinuous movements. To date, however, only discontinuous pacers (e.g., metronomes) have been investigated to compare discontinuous and continuous SMS. We propose an alternative explanation whereby the discontinuous SMS has benefited from the matching between the (dis)continuous nature of the pacer and the (dis)continuous nature of the movements of synchronisation. The present experiment tested this explanation by examining the relative stability of discontinuous and continuous SMS when synchronising with a continuous pacer. Twelve participants finger tapped (discontinuous SMS) or continuously oscillated their forearm (continuous SMS) in synchrony with an oscillatory visual target. The continuity of the pacer was manipulated by varying the kinematic (harmonic to Rayleigh-like oscillations) and the frequency (0.5 and 1Hz) of the target oscillations. Overall, the results showed a more stable continuous than discontinuous SMS. Furthermore, the stability of the discontinuous SMS improved when increasing the discontinuity of the target displacements (high nonlinear kinematic and low frequency), showing an interaction between movement type and pacer continuity in SMS.