Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research

A modular machine learning tool for holistic and fine-grained behavioral analysis

Artificial intelligence techniques offer promising avenues for exploring human body features from videos, yet no freely accessible tool has reliably provided holistic and fine-grained behavioral analyses to date. To address this, we developed a machine learning tool based on a two-level approach: a first lower-level processing using computer vision for extracting fine-grained and comprehensive behavioral features such as skeleton or facial points, gaze, and action units; a second level of machine learning classification coupled with explainability providing modularity, to determine which behavioral features are triggered by specific environments. To validate our tool, we filmed 16 participants across six conditions, varying according to the presence of a person ("Pers"), a sound ("Snd"), or silence ("Rest"), and according to emotional levels using self-referential ("Self") and control ("Ctrl") stimuli. We demonstrated the effectiveness of our approach by extracting and correcting behavior from videos using two computer vision software (OpenPose and OpenFace) and by training two algorithms (XGBoost and long short-term memory [LSTM]) to differentiate between experimental conditions. High classification rates were achieved for "Pers" conditions versus "Snd" or "Rest" (AUC = 0.8-0.9), with explainability revealing actions units and gaze as key features. Additionally, moderate classification rates were attained for "Snd" versus "Rest" (AUC = 0.7), attributed to action units, limbs and head points, as well as for "Self" versus "Ctrl" (AUC = 0.7-0.8), due to facial points. These findings were consistent with a more conventional hypothesis-driven approach. Overall, our study suggests that our tool is well suited for holistic and fine-grained behavioral analysis and offers modularity for extension into more complex naturalistic environments.

Self-injury and suicidal behaviors in high-risk adolescents: Distal predictors, proximal correlates, and interactive effects of impulsivity and emotion dysregulation

Suicide rates are rising among U.S. youth, yet our understanding of developmental mechanisms associated with increased suicide risk is limited. One high-risk pathway involves an interaction between heritable trait impulsivity and emotion dysregulation (ED). Together, these confer increased vulnerability to nonsuicidal self-injury (NSSI), suicide ideation (SI), and suicide attempts (SAs). Previous work, however, has been limited to homogeneous samples. We extend the Impulsivity × ED hypothesis to a more diverse sample of adolescents (N = 344, ages 12-15 at Baseline, 107 males and 237 females) who were treated for major depression and assessed four times over two years. In multilevel models, the impulsivity × ED interaction was associated with higher levels and worse trajectories of NSSI, SI, and SAs. As expected, stressful life events were also associated with poorer trajectories for all outcomes, and NSSI was associated with future and concurrent SI and SAs. These findings extend one developmental pathway of risk for self-harming and suicidal behaviors to more diverse adolescents, with potential implications for prevention.

Impaired facial emotion recognition in individuals with bipolar disorder

BACKGROUND

Individuals with bipolar disorder (BD) often struggle with emotional regulation and social interactions, partly due to difficulties in accurately recognizing facial emotions.

METHODS

From September 2021 to February 2023, 69 BD individuals-comprising 23 with bipolar manic/hypomanic episode (BME), 23 with bipolar depressive episode (BDE), 23 with bipolar euthymic (EUT)-and 23 healthy controls (HCs) were enrolled. Diagnosis adhered to DSM-IV criteria using M.I.N.I 5.0, alongside assessments via Hamilton Depression Scale 17 and Young Manic Rating Scale. Recognition tasks involving 84 facial expression images across six categories. The Wilcoxon rank-sum test compares two groups, while the Kruskal-Wallis test compares multiple groups with subsequent adjusted pairwise comparisons.

RESULTS

The overall correct recognition rate of facial expressions in the BD group (79 %) was significantly lower than that of the HC group (83 %) (P=0.004). Primary differences were noted in neutral (93 % vs. 100 %, P=0.012) and fear (79 % vs. 86 %, P=0.023) expressions. Within the BD group, correct recognition rates were 71 % for BME, 80 % for BDE, and 80 % for EUT, all lower than in the HC group. Significant differences in correct recognition rates of neutral, fear, and joy expressions were observed among the four groups (P<0.05), with the BME group exhibiting the lowest rate. Misidentification of facial expressions was more frequent in the BD group compared to the HC group, particularly among negative expressions.

CONCLUSION

Patients with BD demonstrate lower correct recognition and higher misidentification rates of facial expressions, with those experiencing manic episodes showing impaired recognition of neutral, joy, and fear expressions.

Infrequent facial expressions of emotion do not bias attention

Despite the obvious importance of facial expressions of emotion, most studies have found that they do not bias attention. A critical limitation, however, is that these studies generally present face distractors on all trials of the experiment. For other kinds of emotional stimuli, such as emotional scenes, infrequently presented stimuli elicit greater attentional bias than frequently presented stimuli, perhaps due to suppression or habituation. The goal of the current study then was to test whether such modulation of attentional bias by distractor frequency generalizes to facial expressions of emotion. In Experiment 1, both angry and happy faces were unable to bias attention, despite being infrequently presented. Even when the location of these face cues were more unpredictable-presented in one of two possible locations-still no attentional bias was observed (Experiment 2). Moreover, there was no bottom-up influence for angry and happy faces shown under high or low perceptual load (Experiment 3). We conclude that task-irrelevant posed facial expressions of emotion cannot bias attention even when presented infrequently.

Quantifying the efficacy of an automated facial coding software using videos of parents

Introduction

This work explores the use of an automated facial coding software - FaceReader - as an alternative and/or complementary method to manual coding.

Methods

We used videos of parents (fathers, n = 36; mothers, n = 29) taken from the Avon Longitudinal Study of Parents and Children. The videos-obtained during real-life parent-infant interactions in the home-were coded both manually (using an existing coding scheme) and by FaceReader. We established a correspondence between the manual and automated coding categories - namely Positive, Neutral, Negative, and Surprise - before contingency tables were employed to examine the software's detection rate and quantify the agreement between manual and automated coding. By employing binary logistic regression, we examined the predictive potential of FaceReader outputs in determining manually classified facial expressions. An interaction term was used to investigate the impact of gender on our models, seeking to estimate its influence on the predictive accuracy.

Results

We found that the automated facial detection rate was low (25.2% for fathers, 24.6% for mothers) compared to manual coding, and discuss some potential explanations for this (e.g., poor lighting and facial occlusion). Our logistic regression analyses found that Surprise and Positive expressions had strong predictive capabilities, whilst Negative expressions performed poorly. Mothers' faces were more important for predicting Positive and Neutral expressions, whilst fathers' faces were more important in predicting Negative and Surprise expressions.

Discussion

We discuss the implications of our findings in the context of future automated facial coding studies, and we emphasise the need to consider gender-specific influences in automated facial coding research.

Coding infant engagement in the Face-to-Face Still-Face paradigm using deep neural networks

BACKGROUND

The Face-to-Face Still-Face (FFSF) task is a validated and commonly used observational measure of mother-infant socio-emotional interactions. With the ascendence of deep learning-based facial emotion recognition, it is possible that common complex tasks, such as the coding of FFSF videos, could be coded with a high degree of accuracy by deep neural networks (DNNs). The primary objective of this study was to test the accuracy of four DNN image classification models against the coding of infant engagement conducted by two trained independent manual raters.

METHODS

68 mother-infant dyads completed the FFSF task at three timepoints. Two trained independent raters undertook second-by-second manual coding of infant engagement into one of four classes: 1) positive affect, 2) neutral affect, 3) object/environment engagement, and 4) negative affect.

RESULTS

Training four different DNN models on 40,000 images, we achieved a maximum accuracy of 99.5% on image classification of infant frames taken from recordings of the FFSF task with a maximum inter-rater reliability (Cohen's κ-value) of 0.993.

LIMITATIONS

This study inherits all sampling and experimental limitations of the original study from which the data was taken, namely a relatively small and primarily White sample.

CONCLUSIONS

Based on the extremely high classification accuracy, these findings suggest that DNNs could be used to code infant engagement in FFSF recordings. DNN image classification models may also have the potential to improve the efficiency of coding all observational tasks with applications across multiple fields of human behavior research.

Dynamics of Affective Reactivity during Mother-Daughter Interactions: The Impact of Adolescent Non-Suicidal Self-Injury

Non-suicidal self-injury (NSSI) is an alarming public health concern that is particularly widespread among adolescents. The current study examined affective responses during mother-daughter interactions in adolescent girls with and without a history of NSSI. Participants were 60 girls aged 13-17 with (n = 27) and without (n = 33) a history of NSSI and their mothers. Adolescents and their mothers completed two interaction tasks: one positive and one negative. During these interactions, facial affect was assessed via electromyography (EMG). Results of Actor-Partner Interdependence Modeling (APIM) revealed several intra- and interpersonal disruptions in affect during both tasks among dyads in which the adolescent had an NSSI history. Findings suggest deficits in both self- and co-regulation of facial affect during mother-daughter interactions involving dyads in which the adolescents reports NSSI. Ultimately, if replicated and extended in longitudinal research, these disruptions may prove to be promising targets of intervention to reduce risk for future NSSI in adolescent girls.

Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism

Commonly used screening tools for autism spectrum disorder (ASD) generally rely on subjective caregiver questionnaires. While behavioral observation is more objective, it is also expensive, time-consuming, and requires significant expertise to perform. As such, there remains a critical need to develop feasible, scalable, and reliable tools that can characterize ASD risk behaviors. This study assessed the utility of a tablet-based behavioral assessment for eliciting and detecting one type of risk behavior, namely, patterns of facial expression, in 104 toddlers (ASD N = 22) and evaluated whether such patterns differentiated toddlers with and without ASD. The assessment consisted of the child sitting on his/her caregiver's lap and watching brief movies shown on a smart tablet while the embedded camera recorded the child's facial expressions. Computer vision analysis (CVA) automatically detected and tracked facial landmarks, which were used to estimate head position and facial expressions (Positive, Neutral, All Other). Using CVA, specific points throughout the movies were identified that reliably differentiate between children with and without ASD based on their patterns of facial movement and expressions (area under the curves for individual movies ranging from 0.62 to 0.73). During these instances, children with ASD more frequently displayed Neutral expressions compared to children without ASD, who had more All Other expressions. The frequency of All Other expressions was driven by non-ASD children more often displaying raised eyebrows and an open mouth, characteristic of engagement/interest. Preliminary results suggest computational coding of facial movements and expressions via a tablet-based assessment can detect differences in affective expression, one of the early, core features of ASD. LAY SUMMARY: This study tested the use of a tablet in the behavioral assessment of young children with autism. Children watched a series of developmentally appropriate movies and their facial expressions were recorded using the camera embedded in the tablet. Results suggest that computational assessments of facial expressions may be useful in early detection of symptoms of autism.

Emerging Opportunities Provided by Technology to Advance Research in Child Health Globally

Current approaches to longitudinal assessment of children's developmental and psychological well-being, as mandated in the United Nations Sustainable Development Goals, are expensive and time consuming. Substantive understanding of global progress toward these goals will require a suite of new robust, cost-effective research tools designed to assess key developmental processes in diverse settings. While first steps have been taken toward this end through efforts such as the National Institutes of Health's Toolbox, experience-near approaches including naturalistic observation have remained too costly and time consuming to scale to the population level. This perspective presents 4 emerging technologies with high potential for advancing the field of child health and development research, namely (1) affective computing, (2) ubiquitous computing, (3) eye tracking, and (4) machine learning. By drawing attention of scientists, policy makers, investors/funders, and the media to the applications and potential risks of these emerging opportunities, we hope to inspire a fresh wave of innovation and new solutions to the global challenges faced by children and their families.

Technology Use for Adolescent Health and Wellness

As avid users of technology, adolescents are a key demographic to engage when designing and developing technology applications for health. There are multiple opportunities for improving adolescent health, from promoting preventive behaviors to providing guidance for adolescents with chronic illness in supporting treatment adherence and transition to adult health care systems. This article will provide a brief overview of current technologies and then highlight new technologies being used specifically for adolescent health, such as artificial intelligence, virtual and augmented reality, and machine learning. Because there is paucity of evidence in this field, we will make recommendations for future research.