Head Movement Dynamics During Play and Perturbed Mother-Infant Interaction

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

Predicting Autism from Head Movement Patterns during Naturalistic Social Interactions

Autism spectrum disorder (ASD) is a neurodevelopmental condition characterized in part by difficulties in verbal and nonverbal social communication. Evidence indicates that autistic people, compared to neurotypical peers, exhibit differences in head movements, a key form of nonverbal communication. Despite the crucial role of head movements in social communication, research on this nonverbal cue is relatively scarce compared to other forms of nonverbal communication, such as facial expressions and gestures. There is a need for scalable, reliable, and accurate instruments for measuring head movements directly within the context of social interactions. In this study, we used computer vision and machine learning to examine the head movement patterns of neurotypical and autistic individuals during naturalistic, face-to-face conversations, at both the individual (monadic) and interpersonal (dyadic) levels. Our model predicts diagnostic status using dyadic head movement data with an accuracy of 80%, highlighting the value of head movement as a marker of social communication. The monadic data pipeline had lower accuracy (69.2%) compared to the dyadic approach, emphasizing the importance of studying back-and-forth social communication cues within a true social context. The proposed classifier is not intended for diagnostic purposes, and future research should replicate our findings in larger, more representative samples.

Infant AFAR: Automated facial action recognition in infants

Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.

Head Movement Patterns during Face-to-Face Conversations Vary with Age

Advances in computational behavior analysis have the potential to increase our understanding of behavioral patterns and developmental trajectories in neurotypical individuals, as well as in individuals with mental health conditions marked by motor, social, and emotional difficulties. This study focuses on investigating how head movement patterns during face-to-face conversations vary with age from childhood through adulthood. We rely on computer vision techniques due to their suitability for analysis of social behaviors in naturalistic settings, since video data capture can be unobtrusively embedded within conversations between two social partners. The methods in this work include unsupervised learning for movement pattern clustering, and supervised classification and regression as a function of age. The results demonstrate that 3-minute video recordings of head movements during conversations show patterns that distinguish between participants that are younger vs. older than 12 years with 78% accuracy. Additionally, we extract relevant patterns of head movement upon which the age distinction was determined by our models.

Narrative as co-regulation: A review of embodied narrative in infant development

We review evidence of non-verbal, embodied narratives in human infancy to better understand their form and function as generators of common experience, regulation, and learning. We examine their development prior to the onset of language, with a view to improve understanding of narrative as regular motifs or schemas of early experience in both solitary and social engagement. Embodied narratives are composed of regular patterns of interest, arousal, affect, and intention that yield a characteristic four-part structure of (i) introduction, (ii) development, (iii) climax, and (iv) resolution. Made with others these form co-created shared acts of meaning, and are parsed in time with discreet beginnings and endings that allow a regular pattern to frame and give predictive understanding for prospective regulation (especially important within social contexts) that safely returns to baseline again. This characteristic pattern, co-created between infant and adult from the beginning of life, allows the infant to contribute to, and learn, the patterns of its culture. We conclude with a view on commonalities and differences of co-created narrative in non-human primates, and discuss implications of disruption to narrative co-creation for developmental psychopathology.

Long-term ecological assessment of intracranial electrophysiology synchronized to behavioral markers in obsessive-compulsive disorder

Detection of neural signatures related to pathological behavioral states could enable adaptive deep brain stimulation (DBS), a potential strategy for improving efficacy of DBS for neurological and psychiatric disorders. This approach requires identifying neural biomarkers of relevant behavioral states, a task best performed in ecologically valid environments. Here, in human participants with obsessive-compulsive disorder (OCD) implanted with recording-capable DBS devices, we synchronized chronic ventral striatum local field potentials with relevant, disease-specific behaviors. We captured over 1,000 h of local field potentials in the clinic and at home during unstructured activity, as well as during DBS and exposure therapy. The wide range of symptom severity over which the data were captured allowed us to identify candidate neural biomarkers of OCD symptom intensity. This work demonstrates the feasibility and utility of capturing chronic intracranial electrophysiology during daily symptom fluctuations to enable neural biomarker identification, a prerequisite for future development of adaptive DBS for OCD and other psychiatric disorders.

A Person- and Time-Varying Vector Autoregressive Model to Capture Interactive Infant-Mother Head Movement Dynamics

Head movement is an important but often overlooked component of emotion and social interaction. Examination of regularity and differences in head movements of infant-mother dyads over time and across dyads can shed light on whether and how mothers and infants alter their dynamics over the course of an interaction to adapt to each others. One way to study these emergent differences in dynamics is to allow parameters that govern the patterns of interactions to change over time, and according to person- and dyad-specific characteristics. Using two estimation approaches to implement variations of a vector-autoregressive model with time-varying coefficients, we investigated the dynamics of automatically-tracked head movements in mothers and infants during the Face-Face/Still-Face Procedure (SFP) with 24 infant-mother dyads. The first approach requires specification of a confirmatory model for the time-varying parameters as part of a state-space model, whereas the second approach handles the time-varying parameters in a semi-parametric ("mostly" model-free) fashion within a generalized additive modeling framework. Results suggested that infant-mother head movement dynamics varied in time both within and across episodes of the SFP, and varied based on infants' subsequently-assessed attachment security. Code for implementing the time-varying vector-autoregressive model using two R packages, dynr and mgcv, is provided.

Mother-Infant Interaction Kinect Analysis (MIIKA): An automatic kinematic-based methodology for the investigation of interpersonal distance during early exchanges

Interpersonal distance is a core aspect of mother-child interaction. While conventional measures based on human coders do not fully capture the dynamics of this feature, computational methods provide automatic measures which can detect even small changes and more accurate estimates both spatially and temporally. Using RGB-D sensors (Microsoft Kinect V2), the present study describes a setup to automatically examine interpersonal distance during mother-child interactions, termed Mother-Infant Interaction Kinect Analysis (MIIKA). First, the laboratory setting and the data extraction method are described. By using an ad-hoc algorithm for kinematic data extraction, MIIKA returns three metrics: barycenter position (distance and velocity of approach and separation), movements (number of small, medium and large approaches and separations) and contributions (proportional contributions of mother and child to approaches and separations). Secondly, preliminary MIIKA metrics are described for a non-clinical mother-child dyad as an exemplification of the protocol. As interpersonal distance can be affected by contingent situations, we detected mother-infant full skeleton during three interactional contexts characterized by different kinds of dyadic exchanges: a free play session, a task-oriented activity and an emotionally arousing condition. Results highlighted similarities and differences between the three interactional contexts. MIIKA appears to be a promising setup to automatically examine interpersonal distance in early mother-child interactions.

Detection of eye contact with deep neural networks is as accurate as human experts

Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject's looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers.

Behavior and interaction imaging at 9 months of age predict autism/intellectual disability in high-risk infants with West syndrome

Automated behavior analysis are promising tools to overcome current assessment limitations in psychiatry. At 9 months of age, we recorded 32 infants with West syndrome (WS) and 19 typically developing (TD) controls during a standardized mother-infant interaction. We computed infant hand movements (HM), speech turn taking of both partners (vocalization, pause, silences, overlap) and motherese. Then, we assessed whether multimodal social signals and interactional synchrony at 9 months could predict outcomes (autism spectrum disorder (ASD) and intellectual disability (ID)) of infants with WS at 4 years. At follow-up, 10 infants developed ASD/ID (WS+). The best machine learning reached 76.47% accuracy classifying WS vs. TD and 81.25% accuracy classifying WS+ vs. WS-. The 10 best features to distinguish WS+ and WS- included a combination of infant vocalizations and HM features combined with synchrony vocalization features. These data indicate that behavioral and interaction imaging was able to predict ASD/ID in high-risk children with WS.

Infants’ Motor Activity During a Mother–Infant Interaction Alternating Silent and Singing Phases

Infants interact with their caregivers, using visual contact, smiling, vocalizations, and motor activity. Most of the studies on infant motor activity during dyadic interaction provided qualitative microanalyses. Few documented changes in infants’ general motor activity when facing social stimuli, but the movement analyses were carried out over large time windows. Following on from a previous study, we sought to explore rapid variations in motor activity during an interaction between mothers and their 6-month-old infants. The interaction featured short alternating silent and singing phases. Results showed that infants’ head, hand, and foot motor activity was related to their mothers’ behavior. Head movements, in particular, decreased during the songs and increased between them. The interindividual variability was strong at the hand and foot level and a k-mean cluster analysis showed three different group tendencies. Head stillness certainly expressed the infants’ attentional engagement in the interaction. Hand and foot movements appeared to be more variable and, at the foot level, presumably depended on the mother–infant communicative routines. Infants’ general motor activity could be an indicator of the infant’s engagement and reflect the dyad’s communicative routines during interactions.

Dynamics of Face and Head Movement in Infants with and without Craniofacial Microsomia: An Automatic Approach

Supplemental Digital Content is available in the text.

Background:

Craniofacial microsomia (CFM) is a congenital condition associated with malformations of the bone and soft tissue of the face and the facial nerves, all of which have the potential to impair facial expressiveness. We investigated whether CFM-related variation in expressiveness is evident as early as infancy.

Methods:

Participants were 113 ethnically diverse 13-month-old infants (n = 63 cases with CFM and n = 50 unaffected matched controls). They were observed in 2 emotion induction tasks designed to elicit positive and negative effects. Facial and head movement was automatically measured using a computer vision–based approach. Expressiveness was quantified as the displacement, velocity, and acceleration of 49 facial landmarks (eg, lip corners) and head pitch and yaw.

Results:

For both cases and controls, all measures of expressiveness strongly differed between tasks. Case–control differences were limited to infants with microtia plus mandibular hypoplasia and other associated CFM features, which were the most common phenotypes and were characterized by decreased expressiveness relative to control infants.

Conclusions:

Infants with microtia plus mandibular hypoplasia and those with other associated CFM phenotypes were less facially expressive than same-aged peers. Both phenotypes were associated with more severe involvement than microtia alone, suggesting that infants with more severe CFM begin to diverge in expressiveness from controls by age 13 months. Further research is needed to both replicate the current findings and elucidate their developmental implications.

Disentangling the Dyadic Dance: Theoretical, Methodological and Outcomes Systematic Review of Mother-Infant Dyadic Processes

Background: During the last decades, the research on mother-infant dyad has produced a great amount of data, methods and theories, which largely contributed to set a revolution in the way we look at developmental changes during infancy and childhood. Very different constructs depict the different aspects of the "dyadic dance" occurring between a mother and her infant; nonetheless, a comprehensive and consistent systematization of these concepts in a coherent theoretical landscape is still lacking. Aim: In the present work, we aim at disentangling the different theoretical and methodological definitions of 9 dyadic constructs and we highlight their effects on infants' and children developmental outcomes. Methods: A literature search has been conducted on three databases-PubMed, Scopus, Web of Science. Three different reviews are reported here: (1) a review on the theoretical definitions of dyadic constructs; (2) a review of operational definitions, settings and methods of dyadic processes; (3) a systematic review of dyadic processes' outcomes for infants' and children developmental trajectories. Results: Two constructs emerged as wide meta-theoretical concepts (reciprocity and mutuality) and seven described specific processes (attunement, contingency, coordination, matching, mirroring, reparation, synchrony). A global model resuming the relationships among different processes is reported, which highlights the emergence of two specific cycles of dyadic functioning (i.e., matching-mismatching-reparation-synchrony; contingency, coordination, attunement, mirroring). A comprehensive review of the adopted measures is also provided. Finally, all the processes provided significant contributions to infants' behavioral, cognitive, and socio-emotional development during the first 3 years of age, but limited research has been conducted on specific processes (e.g. reparation and mirroring). Conclusion: The present study provides an original research-grounded framework to consider the different nature of mother-infant dyadic processes within a unified dyadic eco-system. Different levels of evidence emerged for the role of diverse mother-infant dyadic processes on infants' and children development. Open questions and future research directions are highlighted.

Distinguish self- and hetero-perceived stress through behavioral imaging and physiological features

Stress reactivity is a complex phenomenon associated to multiple and multimodal expressions. Response to stressors has an obvious survival function and may be seen as an internal regulation to adapt to threat or danger. The intensity of this internal response can be assessed as the self-perception of the stress response. In species with social organization, this response also serves a communicative function, so-called hetero-perception. Our study presents multimodal stress detection assessment - a new methodology combining behavioral imaging and physiological monitoring for analyzing stress from these two perspectives. The system is based on automatic extraction of 39 behavioral (2D+3D video recording) and 62 physiological (Nexus-10 recording) features during a socially evaluated mental arithmetic test. The analysis with machine learning techniques for automatic classification using Support Vector Machine (SVM) show that self-perception and hetero-perception of social stress are both close but different phenomena: self-perception was significantly correlated with hetero-perception but significantly differed from it. Also, assessing stress with SVM through multimodality gave excellent classification results (F1 score values: 0.9±0.012 for hetero-perception and 0.87±0.021 for self-perception). In the best selected feature subsets, we found some common behavioral and physiological features that allow classification of both self- and hetero-perceived stress. However, we also found the contributing features for automatic classifications had opposite distributions: self-perception classification was mainly based on physiological features and hetero-perception was mainly based on behavioral features.

Head movements and postures as pain behavior

Pain assessment can benefit from observation of pain behaviors, such as guarding or facial expression, and observational pain scales are widely used in clinical practice with nonverbal patients. However, little is known about head movements and postures in the context of pain. In this regard, we analyze videos of three publically available datasets. The BioVid dataset was recorded with healthy participants subjected to painful heat stimuli. In the BP4D dataset, healthy participants performed a cold-pressor test and several other tasks (meant to elicit emotion). The UNBC dataset videos show shoulder pain patients during range-of-motion tests to their affected and unaffected limbs. In all videos, participants were sitting in an upright position. We studied head movements and postures that occurred during the painful and control trials by measuring head orientation from video over time, followed by analyzing posture and movement summary statistics and occurrence frequencies of typical postures and movements. We found significant differences between pain and control trials with analyses of variance and binomial tests. In BioVid and BP4D, pain was accompanied by head movements and postures that tend to be oriented downwards or towards the pain site. We also found differences in movement range and speed in all three datasets. The results suggest that head movements and postures should be considered for pain assessment and research. As additional pain indicators, they possibly might improve pain management whenever behavior is assessed, especially in nonverbal individuals such as infants or patients with dementia. However, in advance more research is needed to identify specific head movements and postures in pain patients.

Pre-linguistic infants employ complex communicative loops to engage mothers in social exchanges and repair interaction ruptures

Language has long been identified as a powerful communicative tool among humans. Yet, pre-linguistic communication, which is common in many species, is also used by human infants prior to the acquisition of language. The potential communicational value of pre-linguistic vocal interactions between human infants and mothers has been studied in the past decades. With 120 dyads (mothers and three- or six-month-old infants), we used the classical Still Face Paradigm (SFP) in which mothers interact freely with their infants, then refrain from communication (Still Face, SF), and finally resume play. We employed innovative automated techniques to measure infant and maternal vocalization and pause, and dyadic parameters (infant response to mother, joint silence and overlap) and the emotional component of Infant Directed Speech (e-IDS) throughout the interaction. We showed that: (i) during the initial free play mothers use longer vocalizations and more e-IDS when they interact with older infants and (ii) infant boys exhibit longer vocalizations and shorter pauses than girls. (iii) During the SF and reunion phases, infants show marked and sustained changes in vocalizations but their mothers do not and (iv) mother-infant dyadic parameters increase in the reunion phase. Our quantitative results show that infants, from the age of three months, actively participate to restore the interactive loop after communicative ruptures long before vocalizations show clear linguistic meaning. Thus, auditory signals provide from early in life a channel by which infants co-create interactions, enhancing the mother-infant bond.

Relations between Automatically Extracted Motion Features and the Quality of Mother-Infant Interactions at 4 and 13 Months

Bodily movements are an essential component of social interactions. However, the role of movement in early mother-infant interaction has received little attention in the research literature. The aim of the present study was to investigate the relationship between automatically extracted motion features and interaction quality in mother-infant interactions at 4 and 13 months. The sample consisted of 19 mother-infant dyads at 4 months and 33 mother-infant dyads at 13 months. The coding system Coding Interactive Behavior (CIB) was used for rating the quality of the interactions. Kinetic energy of upper-body, arms and head motion was calculated and used as segmentation in order to extract coarse- and fine-grained motion features. Spearman correlations were conducted between the composites derived from the CIB and the coarse- and fine-grained motion features. At both 4 and 13 months, longer durations of maternal arm motion and infant upper-body motion were associated with more aversive interactions, i.e., more parent-led interactions and more infant negativity. Further, at 4 months, the amount of motion silence was related to more adaptive interactions, i.e., more sensitive and child-led interactions. Analyses of the fine-grained motion features showed that if the mother coordinates her head movements with her infant's head movements, the interaction is rated as more adaptive in terms of less infant negativity and less dyadic negative states. We found more and stronger correlations between the motion features and the interaction qualities at 4 compared to 13 months. These results highlight that motion features are related to the quality of mother-infant interactions. Factors such as infant age and interaction set-up are likely to modify the meaning and importance of different motion features.

Dyssynchrony and perinatal psychopathology impact of child disease on parents-child interactions, the paradigm of Prader Willi syndrom

BACKGROUND

Infant-mother interaction is a set of bidirectional processes, where the baby is not only affected by the influences of his caregiver, but is also at the origin of considerable modifications. The recent discovery of biological correlates of synchrony during interaction validated its crucial value during child development. Here, we focus on the paradigmatic case of Prader-Willi Syndrome (PWS) where early endocrinal dysfunction is associated with severe hypotonia and early feeding disorder. As a consequence, parent-infant interaction is impaired. In a recent study (Tauber et al., 2017), OXT intranasal infusion was able to partially reverse the feeding phenotype, infant's behavior and brain connectivity. This article details the interaction profile found during feeding in these dyads and their improvement after OXT treatment.

METHODS

Eighteen infants (≤6months) with PWS were recruited and hospitalized 9days in a French reference center for PWS where they were treated with a short course of intranasal OXT. Social withdrawal behavior and mother-infant interaction were assessed on videos of feeding before and after treatment using the Alarm Distress Baby (ADBB) Scale and the Coding Interactive Behavior (CIB) Scale. Raters were blind to treatment status.

RESULTS

At baseline, infants with PWS showed hypotonia, low expressiveness of affects, fatigability and poor involvement in the relationship with severe withdrawal. Parents tended to adapt to their child difficulties, but the interaction was perturbed, tense, restricted and frequently intrusive with a forcing component during the feeding situation. After OXT treatment, infants were more alert, less fatigable, more expressive, and had less social withdrawal. They initiated mutual activities and were more engaged in relationships through gaze, behavior, and vocalizations. They had a better global tonicity with better handling. These modifications helped the parents to be more sensitive and the synchrony of the dyad was in a positive transactional spiral.

CONCLUSION

Dys-synchrony can be induced by children's pathology as well as parental pathology with emotional and developmental impact in the both cases. The PWS paradigm shows us the necessity to sustain early parents-child relationship to avoid establishment of a negative transactional pattern of interaction that can impact child's development.

Interaction and behaviour imaging: a novel method to measure mother-infant interaction using video 3D reconstruction

Studying early interaction is essential for understanding development and psychopathology. Automatic computational methods offer the possibility to analyse social signals and behaviours of several partners simultaneously and dynamically. Here, 20 dyads of mothers and their 13-36-month-old infants were videotaped during mother-infant interaction including 10 extremely high-risk and 10 low-risk dyads using two-dimensional (2D) and three-dimensional (3D) sensors. From 2D+3D data and 3D space reconstruction, we extracted individual parameters (quantity of movement and motion activity ratio for each partner) and dyadic parameters related to the dynamics of partners heads distance (contribution to heads distance), to the focus of mutual engagement (percentage of time spent face to face or oriented to the task) and to the dynamics of motion activity (synchrony ratio, overlap ratio, pause ratio). Features are compared with blind global rating of the interaction using the coding interactive behavior (CIB). We found that individual and dyadic parameters of 2D+3D motion features perfectly correlates with rated CIB maternal and dyadic composite scores. Support Vector Machine classification using all 2D-3D motion features classified 100% of the dyads in their group meaning that motion behaviours are sufficient to distinguish high-risk from low-risk dyads. The proposed method may present a promising, low-cost methodology that can uniquely use artificial technology to detect meaningful features of human interactions and may have several implications for studying dyadic behaviours in psychiatry. Combining both global rating scales and computerized methods may enable a continuum of time scale from a summary of entire interactions to second-by-second dynamics.