Finding Structure in Time: Visualizing and Analyzing Behavioral Time Series

Beyond Words: Speech Coordination Linked to Personality and Appraisals

We studied how personality differences and conversation topics predict interpersonal speech coordination, leading/following dynamics, and nonverbal interactional dominance in dyadic conversations. In a laboratory, 100 undergraduate students (50 same-gender dyads) had a 15-min conversation following three topics (introduction/self-disclosure/argumentation). Their speech coordination and turn-taking (speech/silence) dynamics were assessed through nonlinear time-series analyses: Cross-Recurrence Quantification Analysis (CRQA), Diagonal Cross-Recurrence Profiles (DCRP), and Anisotropic-CRQA. From the time series, we extracted five variables to operationalize speech coordination (global and at lag-zero), leading-following dynamics, and asymmetries in the interacting partners' nonverbal interactional dominance. Interaction appraisals were also assessed. Associations between personality traits Extraversion/Agreeableness, speech coordination, and nonverbal interactional dominance were tested using mixed-effects models. Speech coordination and nonverbal interactional dominance differed across conversational topics and peaked during argumentative conversations. Extraversion was associated with increased speech coordination, and nonverbal interactional dominance, especially during the argumentative conversation. During a self-disclosure conversation, Extraversion concordance was associated with more symmetry in turn-taking dynamics. Speech coordination was generally associated with positive post-conversational appraisals such as wanting to meet in the future or liking the conversation partner, especially in extroverted individuals, whereas introverts seemed to value less swift dynamics. High Agreeableness predicted less speech coordination during argumentative conversations, and increased speech coordination (at lag-zero) predicted reduced perceived naturality in agreeable individuals. This may suggest a trade-off between maintaining swift speech dynamics and the natural flow of conversation for individuals high in Agreeableness.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10919-025-00482-3.

A multi-level analysis of motor and behavioural dynamics in 9-month-old preterm and term-born infants during changing emotional and interactive contexts

Computational analysis of infant movement has significant potential to reveal markers of developmental health. We report two studies employing dynamic analyses of motor kinematics and motor behaviours, which characterise movement at two levels, in 9-month-old infants. We investigate the effect of preterm birth (< 33 weeks of gestation) and the effect of changing emotional and social-interactive contexts in the still-face paradigm. First, multiscale permutation entropy was employed to analyse acceleration kinematic timeseries data collected from Inertial Measurement Unit (IMU) sensors on infants' torso, wrists, and ankles (N = 32: 10 term; 22 preterm). Second, Recurrence Quantification Analysis was used to characterise patterns of second-to-second behavioural changes, from observationally coded behavioural timeseries on infants' emotional self-regulation (N = 111: 61 term; 50 preterm). We found frequency-specific effects of context on permutation entropy. Relative to infants born at term (> 37 weeks of gestation), infants born preterm showed greater permutation entropy in their left ankle and torso movements, but not in right ankle or wrist movements. We did not find effects of preterm birth or emotional context on micro-level behavioural dynamics. Our methodology and findings inform future work using multiscale entropy to study infant development. Dynamic analysis of behaviour is a relatively young field, and applications to emotional self-regulation requires further methodological development.

HuBar: A Visual Analytics Tool to Explore Human Behavior Based on fNIRS in AR Guidance Systems

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach through an in-depth examination of a think-aloud experiment with five domain experts.

A scoping review of interaction dynamics in minimally verbal autistic individuals

Interaction dynamics provide information about how social interactions unfold over time and have implications for communication development. Characterizing social interaction in autistic people who are minimally verbal (MV) has the potential to illuminate mechanisms of change in communication development and intervention. The purpose of this scoping review was to investigate the current evidence characterizing interaction dynamics in MV autistic individuals, methods used to measure interaction dynamics in this population, and opportunities for future research. Articles were included if participants were diagnosed with autism, considered MV, if interaction occurred with a human communication partner during live in-person interaction, and if variables were derived by measuring the relationship between behaviors in both partners. The seven articles included in this review demonstrate that limited research describes interaction dynamics in this population, and that behavioral coding measures can be leveraged to assess constructs such as turn-taking, social contingency, and balance in social interactions. While there is some evidence describing how MV autistic individuals and their communication partners construct reciprocal interaction, there is variability in how interaction dynamics are measured and limited evidence describing individual differences. Recommendations for future research are discussed.

Early social-cognitive development as a dynamic developmental system-a lifeworld approach

Based on developmental systems and dynamic systems theories, we propose the lifeworld approach-a conceptual framework for research and a hypothesis concerning early social-cognitive development. As a framework, the lifeworld approach recognizes the social embeddedness of development and shifts the focus away from individual developmental outcomes toward the reciprocal interplay of processes within and between individuals that co-constitutes early social-cognitive development. As a hypothesis, the lifeworld approach proposes that the changing developmental system-spanning the different individuals as their subsystems-strives toward attractor states through regulation at the behavioral level, which results in both the emergence and further differentiation of developmental attainments. The lifeworld approach-as a framework and a hypothesis, including key methodological approaches to test it-is exemplified by research on infants' self-awareness, prosocial behavior and social learning. Equipped with, first, a conceptual framework grounded in a modern view on development and, second, a growing suite of methodological approaches, developmental science can advance by analyzing the mutually influential relations between intra-individual and interactional processes in order to identify key mechanisms underlying early social-cognitive development.

An automated, data-driven approach to children's social dynamics in space and time

Most children first enter social groups of peers in preschool. In this context, children use movement as a social tool, resulting in distinctive proximity patterns in space and synchrony with others over time. However, the social implications of children's movements with peers in space and time are difficult to determine due to the difficulty of acquiring reliable data during natural interactions. In this article, we review research demonstrating that proximity and synchrony are important indicators of affiliation among preschoolers and highlight challenges in this line of research. We then argue for the advantages of using wearable sensor technology and machine learning analytics to quantify social movement. This technological and analytical advancement provides an unprecedented view of complex social interactions among preschoolers in natural settings, and can help integrate young children's movements with others in space and time into a coherent interaction framework.

Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama - a signal-analysis approach

A mere co-presence of an unfamiliar person may modulate an individual's attentive engagement with specific events or situations to a significant degree. To understand better how such social presence affects experiences, we recorded a set of parallel multimodal facial and psychophysiological data with subjects (N = 36) who listened to dramatic audio scenes alone or when facing an unfamiliar person. Both a selection of 6 s affective sound clips (IADS-2) followed by a 27 min soundtrack extracted from a Finnish episode film depicted familiar and often intense social situations familiar from the everyday world. Considering the systemic complexity of both the chosen naturalistic stimuli and expected variations in the experimental social situation, we applied a novel combination of signal analysis methods using inter-subject correlation (ISC) analysis, Representational Similarity Analysis (RSA) and Recurrence Quantification Analysis (RQA) followed by gradient boosting classification. We report our findings concerning three facial signals, gaze, eyebrow and smile that can be linked to socially motivated facial movements. We found that ISC values of pairs, whether calculated on true pairs or any two individuals who had a partner, were lower than the group with single individuals. Thus, audio stimuli induced more unique responses in those subjects who were listening to it in the presence of another person, while individual listeners tended to yield a more uniform response as it was driven by dramatized audio stimulus alone. Furthermore, our classifiers models trained using recurrence properties of gaze, eyebrows and smile signals demonstrated distinctive differences in the recurrence dynamics of signals from paired subjects and revealed the impact of individual differences on the latter. We showed that the presence of an unfamiliar co-listener that modifies social dynamics of dyadic listening tasks can be detected reliably from visible facial modalities. By applying our analysis framework to a broader range of psycho-physiological data, together with annotations of the content, and subjective reports of participants, we expected more detailed dyadic dependencies to be revealed. Our work contributes towards modeling and predicting human social behaviors to specific types of audio-visually mediated, virtual, and live social situations.

Eye blinking, musical processing, and subjective states-A methods account

Affective sciences often make use of self-reports to assess subjective states. Seeking a more implicit measure for states and emotions, our study explored spontaneous eye blinking during music listening. However, blinking is understudied in the context of research on subjective states. Therefore, a second goal was to explore different ways of analyzing blink activity recorded from infra-red eye trackers, using two additional data sets from earlier studies differing in blinking and viewing instructions. We first replicate the effect of increased blink rates during music listening in comparison with silence and show that the effect is not related to changes in self-reported valence, arousal, or to specific musical features. Interestingly, but in contrast, felt absorption reduced participants' blinking. The instruction to inhibit blinking did not change results. From a methodological perspective, we make suggestions about how to define blinks from data loss periods recorded by eye trackers and report a data-driven outlier rejection procedure and its efficiency for subject-mean analyses, as well as trial-based analyses. We ran a variety of mixed effects models that differed in how trials without blinking were treated. The main results largely converged across accounts. The broad consistency of results across different experiments, outlier treatments, and statistical models demonstrates the reliability of the reported effects. As recordings of data loss periods come for free when interested in eye movements or pupillometry, we encourage researchers to pay attention to blink activity and contribute to the further understanding of the relation between blinking, subjective states, and cognitive processing.

Integrating cognition in the laboratory with cognition in the real world: the time cognition takes, task fidelity, and finding tasks when they are mixed together

It is now possible for real-life activities, unfolding over their natural range of temporal and spatial scales, to become the primary targets of cognitive studies. Movement toward this type of research will require an integrated methodological approach currently uncommon in the field. When executed hand in hand with thorough and ecologically valid empirical description, properly developed laboratory tasks can serve as model systems to capture the essentials of a targeted real-life activity. When integrated together, data from these two kinds of studies can facilitate causal analysis and modeling of the mental and neural processes that govern that activity, enabling a fuller account than either method can provide on its own. The resulting account, situated in the activity's natural environmental, social, and motivational context, can then enable effective and efficient development of interventions to support and improve the activity as it actually unfolds in real time. We believe that such an integrated multi-level research program should be common rather than rare and is necessary to achieve scientifically and societally important goals. The time is right to finally abandon the boundaries that separate the laboratory from the outside world.

Intentional Understanding Through Action Coordination in Early Triadic Interactions

The ability to understand the behaviour of other people in intentional terms has been traditionally explained by resorting to inferential mechanisms that would allow individuals to access the internal mental states of others. In recent years, the second-person perspective has established itself as a theoretical alternative to traditional models. It argues that intentional understanding is an embodied, natural, and immediate process that occurs in situations such as face-to-face early dyadic interactions between adults and infants. In this article, we argue that the way in which the second-person perspective regards body and object is problematic. Based on psychological evidence that demonstrates the constitutive role of the body and objects for cognitive development, we propose the foundations of an ecological-enactive, semiotic and pragmatic model of intentional understanding. We argue that intentional understanding should be conceived as the skilful coordination of behaviours that subjects come to enact in interactive settings, following the dynamics of bodily and material practices that have acquired normative force over time.

Parent-infant affect synchrony during social and solo play

While mother-infant affect synchrony has been proposed to facilitate the early development of social understanding, most investigations into affect synchrony have concentrated more on negative than positive affect. We analysed affect sharing during parent-infant object play, comparing positive and negative affect, to examine how it is modulated by shared playful activity. Mother-infant dyads (N = 20, average infant age 10.7 months) played together (social) or separately (solo) using an object. Both participants increased positive affect during social play as compared with solo play. Positive affect synchrony also increased during social play compared with solo play, whereas negative affect synchrony did not differ. Closer examination of the temporal dynamics of affect changes showed that infants' shifts to positive affect tended to occur contingently in response to their mothers', whereas mothers' shifts to negative affect followed their infants'. Further, during social play, positive affect displays were more long-lived while negative more short-lived. While our sample was small and from a homogeneous population (e.g. white, highly educated parents), limiting the implications of the findings, these results demonstrate that maternal active engagement in playful interaction with her infant affords, increases, and extends infant positive affect and parent-infant positive affect synchrony, providing insights into how the social context modulates infants' affective experiences. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

The road to conventional tool use: Developmental changes in children's material engagement with artifacts in nursery school

The development of tool use in early childhood is a topic of continuing interest in developmental psychology. However, the lack of studies in ecological settings results in many unknowns about how children come to use artifacts according to their cultural function. We report a longitudinal study with 17 sociodemographically diverse children (8 female) attending a nursery school in Madrid (Spain) and their two adult female teachers. Using mixed-effects models and Granger causality analysis, we measured changes in the frequency and duration of children's object uses between 7 and 17 months of age and in the directional influences among pairs of behaviors performed by teachers and children. Results show a clear shift in how children use artifacts. As early as 12 months of age, the frequency of conventional uses outweighs that of all other types of object use. In addition, object uses become shorter in duration with age, irrespective of their type. Moreover, certain teachers' nonlinguistic communicative strategies (e.g., demonstrations of canonical use and placing gestures) significantly influence and promote children's conventional tool use. Findings shed light on how children become increasingly proficient in conventional tool use through interactions with artifacts and others in nursery school.

Sampling statistics are like story creation: a network analysis of parent-toddler exploratory play

Actions in the world elicit data for learning and do so in a stream of interconnected events. Here, we provide evidence on how toddlers with their parent sample information by acting on toys during exploratory play. We observed 10 min of free-flowing and unconstrained object exploration of by toddlers (mean age 21 months) and parents in a room with many available objects (n = 32). Borrowing concepts and measures from the study of narratives, we found that the toy selections are not a string of unrelated events but exhibit a suite of what we call coherence statistics: Zipfian distributions, burstiness and a network structure. We discuss the transient memory processes that underlie the moment-to-moment toy selections that create this coherence and the role of these statistics in the development of abstract and generalizable systems of knowledge. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.

Allostasis and metastasis: The yin and yang of childhood self-regulation

Most research has studied self-regulation by presenting experimenter-controlled test stimuli and measuring change between baseline and stimulus. In the real world, however, stressors do not flash on and off in a predetermined sequence, and there is no experimenter controlling things. Rather, the real world is continuous and stressful events can occur through self-sustaining interactive chain reactions. Self-regulation is an active process through which we adaptively select which aspects of the social environment we attend to from one moment to the next. Here, we describe this dynamic interactive process by contrasting two mechanisms that underpin it: the "yin" and "yang" of self-regulation. The first mechanism is allostasis, the dynamical principle underlying self-regulation, through which we compensate for change to maintain homeostasis. This involves upregulating in some situations and downregulating in others. The second mechanism is metastasis, the dynamical principle underling dysregulation. Through metastasis, small initial perturbations can become progressively amplified over time. We contrast these processes at the individual level (i.e., examining moment-to-moment change in one child, considered independently) and also at the inter-personal level (i.e., examining change across a dyad, such as a parent-child dyad). Finally, we discuss practical implications of this approach in improving the self-regulation of emotion and cognition, in typical development and psychopathology.

Parliamentary roll-call voting as a complex dynamical system: The case of Chile

A method is proposed to study the temporal variability of legislative roll-call votes in a parliament from the perspective of complex dynamical systems. We studied the Chilean Chamber of Deputies' by analyzing the agreement ratio and the voting outcome of each vote over the last 19 years with a Recurrence Quantification Analysis and an entropy analysis (Sample Entropy). Two significant changes in the temporal variability were found: one in 2014, where the voting outcome became more recurrent and with less entropy, and another in 2018, where the agreement ratio became less recurrent and with higher entropy. These changes may be directly related to major changes in the Chilean electoral system and the composition of the Chamber of Deputies, given that these changes occurred just after the first parliamentary elections with non-compulsory voting (2013 elections) and the first elections with a proportional system in conjunction with an increase in the number of deputies (2017 elections) were held.

Time Is of the Essence: Neural Codes, Synchronies, Oscillations, Architectures

Time is of the essence in how neural codes, synchronies, and oscillations might function in encoding, representation, transmission, integration, storage, and retrieval of information in brains. This Hypothesis and Theory article examines observed and possible relations between codes, synchronies, oscillations, and types of neural networks they require. Toward reverse-engineering informational functions in brains, prospective, alternative neural architectures incorporating principles from radio modulation and demodulation, active reverberant circuits, distributed content-addressable memory, signal-signal time-domain correlation and convolution operations, spike-correlation-based holography, and self-organizing, autoencoding anticipatory systems are outlined. Synchronies and oscillations are thought to subserve many possible functions: sensation, perception, action, cognition, motivation, affect, memory, attention, anticipation, and imagination. These include direct involvement in coding attributes of events and objects through phase-locking as well as characteristic patterns of spike latency and oscillatory response. They are thought to be involved in segmentation and binding, working memory, attention, gating and routing of signals, temporal reset mechanisms, inter-regional coordination, time discretization, time-warping transformations, and support for temporal wave-interference based operations. A high level, partial taxonomy of neural codes consists of channel, temporal pattern, and spike latency codes. The functional roles of synchronies and oscillations in candidate neural codes, including oscillatory phase-offset codes, are outlined. Various forms of multiplexing neural signals are considered: time-division, frequency-division, code-division, oscillatory-phase, synchronized channels, oscillatory hierarchies, polychronous ensembles. An expandable, annotative neural spike train framework for encoding low- and high-level attributes of events and objects is proposed. Coding schemes require appropriate neural architectures for their interpretation. Time-delay, oscillatory, wave-interference, synfire chain, polychronous, and neural timing networks are discussed. Some novel concepts for formulating an alternative, more time-centric theory of brain function are discussed. As in radio communication systems, brains can be regarded as networks of dynamic, adaptive transceivers that broadcast and selectively receive multiplexed temporally-patterned pulse signals. These signals enable complex signal interactions that select, reinforce, and bind common subpatterns and create emergent lower dimensional signals that propagate through spreading activation interference networks. If memory traces share the same kind of temporal pattern forms as do active neuronal representations, then distributed, holograph-like content-addressable memories are made possible via temporal pattern resonances.

Infant exuberant object play at home: Immense amounts of time-distributed, variable practice

Object play yields enormous benefits for infant development. However, little is known about natural play at home where most object interactions occur. We conducted frame-by-frame video analyses of spontaneous activity in two 2-h home visits with 13-month-old crawling infants and 13-, 18-, and 23-month-old walking infants (N = 40; 21 boys; 75% White). Regardless of age, for every infant and time scale, across 10,015 object bouts, object interactions were short (median = 9.8 s) and varied (transitions among dozens of toys and non-toys) but consumed most of infants' time. We suggest that infant exuberant object play-immense amounts of brief, time-distributed, variable interactions with objects-may be conducive to learning object properties and functions, motor skill acquisition, and growth in cognitive, social, and language domains.

Quantifying Everyday Ecologies: Principles for Manual Annotation of Many Hours of Infants' Lives

Everyday experiences are the experiences available to shape developmental change. Remarkable advances in devices used to record infants' and toddlers' everyday experiences, as well as in repositories to aggregate and share such recordings across teams of theorists, have yielded a potential gold mine of insights to spur next-generation theories of experience-dependent change. Making full use of these advances, however, currently requires manual annotation. Manually annotating many hours of everyday life is a dedicated pursuit requiring significant time and resources, and in many domains is an endeavor currently lacking foundational facts to guide potentially consequential implementation decisions. These realities make manual annotation a frequent barrier to discoveries, as theorists instead opt for narrower scoped activities. Here, we provide theorists with a framework for manually annotating many hours of everyday life designed to reduce both theoretical and practical overwhelm. We share insights based on our team's recent adventures in the previously uncharted territory of everyday music. We identify principles, and share implementation examples and tools, to help theorists achieve scalable solutions to challenges that are especially fierce when annotating extended timescales. These principles for quantifying everyday ecologies will help theorists collectively maximize return on investment in databases of everyday recordings and will enable a broad community of scholars—across institutions, skillsets, experiences, and working environments—to make discoveries about the experiences upon which development may depend.

Parent-Child Joint Behaviors in Novel Object Play Create High-Quality Data for Word Learning

This research takes a dyadic approach to study early word learning and focuses on toddlers' (N = 20, age: 17-23 months) information seeking and parents' information providing behaviors and the ways the two are coupled in real-time parent-child interactions. Using head-mounted eye tracking, this study provides the first detailed comparison of children's and their parents' behavioral and attentional patterns in two free-play contexts: one with novel objects with to-be-learned names (Learning condition) and the other with familiar objects with known names (Play condition). Children and parents in the Learning condition modified their individual and joint behaviors when encountering novel objects with to-be-learned names, which created clearer signals that reduced referential ambiguity and potentially facilitated word learning.

Social synchronization during joint attention in children with autism spectrum disorder

We explored the social synchronization of gaze-shift behaviors when responding to joint attention in children with autism spectrum disorder (ASD). Forty-one children aged 5 to 8 with ASD and 43 typically developing (TD) children watched a video to complete the response to joint attention (RJA) tasks, during which their gaze data were collected. The synchronization of gaze-shift behaviors between children and the female model in the video was measured with the cross-recurrence quantification analysis (CRQA). Ultimately, we discovered that children with ASD had the ability to synchronize their gaze shifts with the female model in the video during RJA tasks. Compared to the TD children, they displayed lower levels of synchronization and longer latency in this synchronized behavior. These findings provide a new avenue to deepen our understanding of the impairments of social interaction in children with ASD. Notably, the analytic method can be further applied to explore the social synchronization of numerous other social interactive behaviors in ASD. LAY SUMMARY: This study explored how autistic children synchronized their gazed shifts with others' gaze cues during joint attention. We found that compared to typical children, autistic children synchronized their gazed shifts less and needed more time to follow others' gaze. These findings provide a new avenue to deepen our understanding of the impairments of social interaction in children with ASD.

"Dancing" Together: Infant-Mother Locomotor Synchrony

Pre-mobile infants and caregivers spontaneously engage in a sequence of contingent facial expressions and vocalizations that researchers have referred to as a social "dance." Does this dance continue when both partners are free to move across the floor? Locomotor synchrony was assessed in 13- to 19-month-old infant-mother dyads (N = 30) by tracking each partner's step-to-step location during free play. Although infants moved more than mothers, dyads spontaneously synchronized their locomotor activity. For 27 dyads, the spatiotemporal path of one partner uniquely identified the path of the other. Clustering analyses revealed two patterns of synchrony (mother-follow and yo-yo), and infants were more likely than mothers to lead the dance. Like face-to-face synchrony, locomotor synchrony scaffolds infants' interactions with the outside world.

Using Vector Autoregression Modeling to Reveal Bidirectional Relationships in Gender/Sex-Related Interactions in Mother-Infant Dyads

Vector autoregression (VAR) modeling allows probing bidirectional relationships in gender/sex development and may support hypothesis testing following multi-modal data collection. We show VAR in three lights: supporting a hypothesis, rejecting a hypothesis, and opening up new questions. To illustrate these capacities of VAR, we reanalyzed longitudinal data that recorded dyadic mother-infant interactions for 15 boys and 15 girls aged 3 to 11 months of age. We examined monthly counts of 15 infant behaviors and 13 maternal behaviors (Seifer et al., 1994). VAR models demonstrated that infant crawling predicted a subsequently close feedback loop from mothers of boys but a subsequently open-ended, branched response from mothers of girls. A different finding showed that boys' standing independently predicted significant later increases of four maternal behaviors: rocking/jiggling, lifting, affectionate touching, and stimulation of infant gross-motor activity. In contrast, crawling by girls led mothers to later decrease the same maternal behaviors. Thus, VAR might allow us to identify how mothers respond differently during daily interactions depending on infant gender/sex. The present work intends to mainly showcase the VAR method in the specific context of the empirical study of gender/sex development.