Associations between Facial Emotion Recognition and Mental Health in Early Adolescence

Hierarchical Context-Based Emotion Recognition With Scene Graphs

For a better intention inference, we often try to figure out the emotional states of other people in social communications. Many studies on affective computing have been carried out to infer emotions through perceiving human states, i.e., facial expression and body posture. Such methods are skillful in a controlled environment. However, it often leads to misestimation due to the deficiency of effective inputs in unconstrained circumstances, that is, where context-aware emotion recognition appeared. We take inspiration from the advanced reasoning pattern of humans in perceived emotion recognition and propose the hierarchical context-based emotion recognition method with scene graphs. We propose to extract three contexts from the image, i.e., the entity context, the global context, and the scene context. The scene context contains abstract information about entity labels and their relationships. It is similar to the information processing of the human visual sensing mechanism. After that, these contexts are further fused to perform emotion recognition. We carried out a bunch of experiments on the widely used context-aware emotion datasets, i.e., CAER-S, EMOTIC, and BOdy Language Dataset (BoLD). We demonstrate that the hierarchical contexts can benefit emotion recognition by improving the accuracy of the SOTA score from 84.82% to 90.83% on CAER-S. The ablation experiments show that hierarchical contexts provide complementary information. Our method improves the F1 score of the SOTA result from 29.33% to 30.24% (C-F1) on EMOTIC. We also build the image-based emotion recognition task with BoLD-Img from BoLD and obtain a better emotion recognition score (ERS) score of 0.2153.

Mental Health of School-Aged Children Treated with Propranolol or Atenolol for Infantile Hemangioma and Their Parents

BACKGROUND

Infants with infantile hemangioma (IH) have been effectively treated with propranolol or atenolol. Concerns were raised about the mental health of these children at school age, due to central nervous system effects of propranolol and visible nature of IH.

OBJECTIVE

This study aimed to compare the mental health at school age of children treated with propranolol to children treated with atenolol for IHs and their parents.

METHODS

This two-centered cross-sectional study included children aged ≥6 years and treated with either propranolol or atenolol for IH during infancy. Children's outcomes were performance-based affect recognition (Dutch version of the Developmental Neuropsychological Assessment-II [NEPSY-II-NL]), parent-reported emotional and behavioral functioning (Child Behavioral Checklist [CBCL]), and health-related quality of life (KIDSCREEN-27). Parents' outcome was parenting stress (Parenting Stress Questionnaire [OBVL]).

RESULTS

Data of 105 children (36 propranolol, 69 atenolol; 6.0-11.8 years) were analyzed. Mental health outcomes did not differ between both β-blocker groups. Although overall functioning was in line with norms, children presented specific problems concerning affect recognition, parent-reported attention, and social quality of life. Parents showed increased physical symptoms, depressive symptoms, and parent-child relationship problems.

CONCLUSION

No difference in mental health at school age was found between children treated with propranolol or atenolol for IH. Although few overall mental health problems were found, specific problems require follow-up. Follow-up of children should be directed toward affect recognition, attention, and social functioning in daily life. Problems reported by parents could be ameliorated by mental health support during and after their infant's β-blocker treatment.

The Association Between Emotion Recognition and Internalizing Problems in Children and Adolescents: A Three-Level Meta-Analysis

Numerous studies have explored the link between how well youth recognize emotions and their internalizing problems, but a consensus remains elusive. This study used a three-level meta-analysis model to quantitatively synthesize the findings of existing studies to assess the relationship. A moderation analysis was also conducted to explore the sources of research heterogeneity. Through a systematic literature search, a total of 42 studies with 201 effect sizes were retrieved for the current meta-analysis, and 7579 participants were included. Emotion recognition was negatively correlated with internalizing problems. Children and adolescents with weaker emotion recognition skills were more likely to have internalizing problems. In addition, this meta-analysis found that publication year had a significant moderating effect. The correlation between emotion recognition and internalizing problems decreased over time. The degree of internalizing problems was also found to be a significant moderator. The correlation between emotion recognition and internalizing disorders was higher than the correlation between emotion recognition and internalizing symptoms. Deficits in emotion recognition might be relevant for the development and/or maintenance of internalizing problems in children and adolescents. The overall effect was small and future research should explore the clinical relevance of the association.

Partner's emotions are associated with preschoolers' prefrontal activation under joint attention: An fNIRS evidence

Emotions and joint attention are highly associated and mutually influenced during preschool, the critical period for early emotional and cognitive development. However, few studies have explored the neuropsychological mechanism of joint attention with preschoolers and their partners under different emotions. This study has examined the prefrontal activation under a comprehensive emotional joint attention task in 45 preschoolers (25 boys, Mage  = 58 ± 9.02 months) to compare the different influences of partners' positive, neutral, and negative emotions. Analysis of the functional near-infrared spectroscopy data indicated that the participants' prefrontal activation triggered by joint attention in positive and negative emotions was significantly higher than in neutral emotions. Moreover, their brain synchronization intensity was significantly higher in positive emotions of joint attention than in negative emotions. These findings advance our understanding of the neural mechanism of early childhood emotional processing under joint attention and provide a neural perspective to explain the effects of different emotions on preschoolers' social cognition.

Analysis of attentional biases in anxiety using 24 facial priming sequences

The processing of emotional facial expressions helps people to adjust to the physical and social environment. Furthermore, mental disorders such as anxiety have been linked to attentional biases in the processing of this type of information. Nevertheless, there are still contradictory results that might be due to the methodology used and to individual differences in the manifestation of anxiety. Our research goal was to use 24 facial priming sequences to analyse attentional biases in the detection of facial expressions of fear, considering the levels and the ways in which individuals express anxiety. With higher levels of cognitive anxiety and general trait anxiety, those sequences that began in the upper half (vs. lower half) elicited a speedier response in the detection of fear. The results are discussed within the context of other techniques and disorders that prompt a deficit in the processing of facial information.

Four attachment-based categories of emotion regulation in adolescent psychic troubles

Introduction

Emotion regulation is altered in many psychiatric disorders in adolescence, but the understanding of mechanisms that underlie this alteration is still poor.

Methods

The PERCEPT study explores alexithymia, empathy, facial emotion recognition (FER) and defence mechanisms in a sample of adolescents in psychiatric care (n = 61, 74% of girls, mean age = 15.03 y.o.), in relation with participants' attachment styles.

Results

Results revealed correlations between attachment dimensions and all of the emotion regulation variables, suggesting that attachment modalities have functional links with emotional regulation at its different levels: FER accuracy was inversely correlated with avoidant attachment, while affective empathy, difficulty in identifying feelings (alexithymia) and immature as well as neurotic defence mechanisms were positively correlated with anxious attachment. Moreover, attachment categories delineated distinct emotional perception profiles. In particular, preoccupied attachment included adolescents with the highest levels of facial emotion perception (sensitivity and accuracy) and of affective empathy, whereas detached attachment included adolescents with the lowest levels of these variables. Neurotic defence mechanisms and difficulty to identify feelings were correlated with preoccupied attachment; immature defence mechanisms and difficulty to describe feelings to others characterized fearful attachment.

Discussion

These results suggest that attachment categories underlie emotion regulation processes in psychiatric disorders in adolescence. Theoretical and clinical implications are discussed.

Facial emotion recognition in adolescent depression: The role of childhood traumas, emotion regulation difficulties, alexithymia and empathy

Introduction

Facial emotion recognition (FER) is crucial for effective social competency, and problems in this skill are linked depression during adolescence. In this study, we aimed to find the rates of FER accuracy for negative (fearful, sad, angry, disgusted), positive (happy, surprised), and neutral emotions, and the possible predictors of FER skill for most confusing emotions.

Subjects and Methods

A total of 67 drug-naive adolescents with depression (11 boys, 56 girls; 11-17 years) were recruited for the study. The facial emotion recognition test, childhood trauma questionnaire and basic empathy, difficulty of emotion regulation, and Toronto alexithymia scales were used.

Results

The analysis demonstrated that adolescents have more difficulties in recognizing negative emotions when compared the positive ones. The most confusing emotion is fear (39.8% of fear was recognized as surprise). Boys have lower fear recognition skill than girls and higher childhood emotional abuse, physical abuse, emotional neglect, and difficulty in describing feelings to predict lower fear recognition skill. For sadness recognition skill, emotional neglect, difficulty in describing feelings, and depression severity were the negative predictors. Emotional empathy has a positive effect on disgust recognition skill.

Conclusion

Our findings demonstrated that impairment of FER skill for negative emotions is associated with childhood traumas, emotion regulation difficulties, alexithymia, and empathy symptoms in adolescent depression.

The influence of childhood emotional neglect on emotional face processing in young adults

Childhood emotional neglect (CEN) refers to a failure to meet the basic emotional needs of a child, which can seriously impact interpersonal communication and psychological health in young adults. Emotional face processing is critical in interpersonal communication; however, whether CEN affects this processing in young adults has not been investigated. Therefore, the current study aimed to explore the effects of CEN on emotional face processing in young adults. Using the Child Trauma Questionnaire, an online survey was conducted with 5010 students from four universities in Tianjin, China. After online interviews and diagnosis by professional doctors, we obtained 20 participants with CEN (CEN group) and 20 without CEN (control group). None of the participants had any mental diseases. A 2 × 4 mixed design was used to investigate the differences in accuracy and response time when identifying the valence of the emotional faces. Compared to the control group, the CEN group identified the valence of all emotional faces more slowly, but there was no significant difference between the two groups in terms of accuracy. CEN caused delayed emotional face processing in young adults, which may be related to unresponsive, unavailable, and limited emotional interaction patterns between parents and their children.

Facial emotion recognition and schizotypal traits: A systematic review of behavioural studies

AIM

Previous research has indicated that individuals expressing high schizotypal traits and patients with Schizotypal Personality Disorder (SPD), show deficits in facial emotion recognition, compared to low schizotypal or control groups. On the other hand, non-significant findings also exist and the association of facial emotion recognition deficits with the different schizotypal dimensions is not well defined, thus limiting any conclusive outcomes. Therefore, the aim of this systematic review was to further clarify this relationship.

METHODS

PsychInfo, Web of Science, Scopus and PubMed were systematically searched, and 23 papers with a cross-sectional design were selected. Nineteen studies examined individuals with high schizotypal traits and four studies evaluated SPD individuals with behavioural facial emotion recognition paradigms and self-report measures or clinical interviews for schizotypal traits. All selected studies were published between 1994 and August 2020.

RESULTS

According to the evidence of studies, high schizotypal individuals and SPD patients have poorer performance in facial emotion recognition tasks. Negative schizotypy was related to lower accuracy for positive and negative emotions and faster emotion labeling while positive schizotypy was associated with worse accuracy for positive, negative and neutral emotions and more biases. Disorganized schizotypy was associated with poorer accuracy for negative emotions and suspiciousness with higher accuracy for disgust faces but lower total accuracy.

CONCLUSIONS

These findings are consistent with the vulnerability for schizophrenia spectrum disorders and support the idea that emotion recognition deficits are trait markers for these conditions. Thus, the effectiveness of early-intervention programmes could increase by also targeting this class of deficits.

A measurement method for mental health based on dynamic multimodal feature recognition

Introduction

The number of college students with mental problems has increased significantly, particularly during COVID-19. However, the clinical features of early-stage psychological problems are subclinical, so the optimal intervention treatment period can easily be missed. Artificial intelligence technology can efficiently assist in assessing mental health problems by mining the deep correlation of multi-dimensional data of patients, providing ideas for solving the screening of normal psychological problems in large-scale college students. Therefore, we propose a mental health assessment method that integrates traditional scales and multimodal intelligent recognition technology to support the large-scale and normalized screening of mental health problems in colleges and universities.

Methods

Firstly, utilize the psychological assessment scales based on human-computer interaction to conduct health questionnaires based on traditional methods. Secondly, integrate machine learning technology to identify the state of college students and assess the severity of psychological problems. Finally, the experiments showed that the proposed multimodal intelligent recognition method has high accuracy and can better proofread normal scale results. This study recruited 1,500 students for this mental health assessment.

Results

The results showed that the incidence of moderate or higher stress, anxiety, and depression was 36.3, 48.1, and 23.0%, which is consistent with the results of our multiple targeted tests.

Conclusion

Therefore, the interactive multimodality emotion recognition method proposed provides an effective way for large-scale mental health screening, monitoring, and intervening in college students' mental health problems.

An Intelligent Mental Health Identification Method for College Students: A Mixed-Method Study

PURPOSE

Mental health assessments that combine patients' facial expressions and behaviors have been proven effective, but screening large-scale student populations for mental health problems is time-consuming and labor-intensive. This study aims to provide an efficient and accurate intelligent method for further psychological diagnosis and treatment, which combines artificial intelligence technologies to assist in evaluating the mental health problems of college students.

MATERIALS AND METHODS

We propose a mixed-method study of mental health assessment that combines psychological questionnaires with facial emotion analysis to comprehensively evaluate the mental health of students on a large scale. The Depression Anxiety and Stress Scale-21(DASS-21) is used for the psychological questionnaire. The facial emotion recognition model is implemented by transfer learning based on neural networks, and the model is pre-trained using FER2013 and CFEE datasets. Among them, the FER2013 dataset consists of 48 × 48-pixel face gray images, a total of 35,887 face images. The CFEE dataset contains 950,000 facial images with annotated action units (au). Using a random sampling strategy, we sent online questionnaires to 400 college students and received 374 responses, and the response rate was 93.5%. After pre-processing, 350 results were available, including 187 male and 153 female students. First, the facial emotion data of students were collected in an online questionnaire test. Then, a pre-trained model was used for emotion recognition. Finally, the online psychological questionnaire scores and the facial emotion recognition model scores were collated to give a comprehensive psychological evaluation score.

RESULTS

The experimental results of the facial emotion recognition model proposed to show that its classification results are broadly consistent with the mental health survey results. This model can be used to improve efficiency. In particular, the accuracy of the facial emotion recognition model proposed in this paper is higher than that of the general mental health model, which only uses the traditional single questionnaire. Furthermore, the absolute errors of this study in the three symptoms of depression, anxiety, and stress are lower than other mental health survey results and are only 0.8%, 8.1%, 3.5%, and 1.8%, respectively.

CONCLUSION

The mixed method combining intelligent methods and scales for mental health assessment has high recognition accuracy. Therefore, it can support efficient large-scale screening of students' psychological problems.

Emotion Recognition Skill in Specific Learning Disorder and Attention-Deficit Hyperactivity Disorder

Objective

Social difficulties can affect both academic and relational-social functioning in common neurodevelopmental disorders such as attention-deficit hyperactivity disorder and specific learning disorder. It is known that social cognitive skills directly affect social functioning. The aim in this study is to examine the social cognitive skill of facial emotion recognition in children with attention-deficit hyperactivity disorder and specific learning disorder and to investigate whether literacy learning is related to emotion recognition.

Methods

In the study, we compared the emotion recognition skill of 41 children with attention-deficit hyperactivity disorder, 50 with specific learning disorder, and 43 typically developed children using the Reading Mind in the Eyes Test. In addition, we evaluated the relationship between children's emotion recognition ability and literacy learning time.

Results

Attention-deficit hyperactivity disorder and specific learning disorder groups did not show a statistically significant difference in terms of Reading Mind in the Eyes Test scores, but both groups had lower scores than their typically eveloped peers (P < .001). In addition, we found that low Reading Mind in the Eyes Test scores were associated with late learning in reading and writing (P < .001, r  = -.033).

Conclusion

This study found that the specific learning disorder and attention-deficit hyperactivity disorder groups showed similar but poorer impact recognition skills than their typically developed peers. In addition, we showed that literacy learning speed is related to emotion recognition.

Voice, Body Cues and Facial Expression in Emotion Recognition of Spanish Children and Adolescents: The Validation of Bell-Lysaker Emotion Recognition Test

Emotion recognition has been traditionally measured trough the recognition of emotional expressions of static faces. Studies suggest that emotion recognition is progressively acquired from early stages in our infancy. However, the literature regarding other emotional domains such as voice or body movements is scarce. Additionally, the number of tools that integrate several domains is limited, especially in children and adolescents, and none of them tested in Spanish samples. Therefore, this study aimed to define the psychometric properties of the Bell-Lysaker Emotion Recognition Task (BLERT) and a new-designed alternate version providing normative data in Spanish children and adolescents (from 8 to 15 years old corresponding to 3th). Moreover, we aim to describe the emotional acquisition trajectory of children and adolescents with a tool that integrates voice, face expressions and body movements. For that purpose, BLERT was translated into Spanish (BLERT-SI) and an alternate version was created (BLERT-SII). A total of 545 children and adolescents from 8 to 15 year-old participated in the study (250 male/295 female). All participants fulfilled BLERT-SI and BLERT-SII within two weeks of difference. Order of presentation was counterbalanced. Results showed that BLERT-SI and SII have good internal consistency (α = .70 and 71 respectively). Test-retest reliability showed a moderate correlation (r = .45; p < .001). Percentages equivalences per age are provided. Age correlated with BLERT-SI (r = .31; p < .001) and BLERT-SII (r = .21; p < .001), showing a progressive acquisition and development of emotion recognition during this period. BLERT-SI and SII are useful tools when studying the follow-up of children and adolescents.

The role of state and trait anxiety in the processing of facial expressions of emotion

State anxiety appears to influence facial emotion processing (Attwood et al. 2017 R. Soc. Open Sci. 4, 160855). We aimed to (i) replicate these findings and (ii) investigate the role of trait anxiety, in an experiment with healthy UK participants (N = 48, 50% male, 50% high trait anxiety). High and low state anxiety were induced via inhalations of 7.5% carbon dioxide enriched air and medical air, respectively. High state anxiety reduced global emotion recognition accuracy (p = 0.01, η p 2 = 0.14 ), but it did not affect interpretation bias towards perceiving anger in ambiguous angry-happy facial morphs (p = 0.18, η p 2 = 0.04 ). We found no clear evidence of a relationship between trait anxiety and global emotion recognition accuracy (p = 0.60, η p 2 = 0.01 ) or interpretation bias towards perceiving anger (p = 0.83, η p 2 = 0.01 ). However, there was greater interpretation bias towards perceiving anger (i.e. away from happiness) during heightened state anxiety, among individuals with high trait anxiety (p = 0.03, d z = 0.33). State anxiety appears to impair emotion recognition accuracy, and among individuals with high trait anxiety, it appears to increase biases towards perceiving anger (away from happiness). Trait anxiety alone does not appear to be associated with facial emotion processing.

Clinician Facial Expression of Emotion Corresponds with Patient Mindset

BACKGROUND

Mental health has a notable and perhaps underappreciated relationship with symptom intensity related to musculoskeletal pathophysiology. Tools for increasing awareness of mental health opportunities may help musculoskeletal specialists identify and address psychological distress and unhealthy misconceptions with greater confidence. One such type of technology-software that identifies emotions by analyzing facial expressions-could be developed as a clinician-awareness tool. A first step in this endeavor is to conduct a pilot study to assess the ability to measure patient mental health through specialist facial expressions.

QUESTIONS/PURPOSES

(1) Does quantification of clinician emotion using facial recognition software correlate with patient psychological distress and unhealthy misconceptions? (2) Is there a correlation between clinician facial expressions of emotions and a validated measure of the quality of the patient-clinician relationship?

METHODS

In a cross-sectional pilot study, between April 2019 and July 2019, we made video recordings of the clinician's face during 34 initial musculoskeletal specialist outpatient evaluations. There were 16 men and 18 women, all fluent and literate in English, with a mean age of 43 ± 15 years. Enrollment was performed according to available personnel, equipment, and room availability. We did not track declines, but there were only a few. Video recordings were analyzed using facial-emotional recognition software, measuring the proportion of time spent by clinicians expressing measured emotions during a consultation. After the visit, patients completed a demographic questionnaire and measures of health anxiety (the Short Health Anxiety Inventory), fear of painful movement (the Tampa Scale for Kinesiophobia), catastrophic or worst-case thinking about pain (the Pain Catastrophizing Scale), symptoms of depression (the Patient Health Questionnaire), and the patient's perception of the quality of their relationship with the clinician (Patient-Doctor Relationship Questionnaire).

RESULTS

Clinician facial expressions consistent with happiness were associated with less patient health anxiety (r = -0.59; p < 0.001) and less catastrophic thinking (r = -0.37; p = 0.03). Lower levels of clinician expressions consistent with sadness were associated with less health anxiety (r = 0.36; p = 0.04), fewer symptoms of generalized anxiety (r = 0.36; p = 0.03), and less catastrophic thinking (r = 0.33; p = 0.05). Less time expressing anger was associated with greater health anxiety (r = -0.37; p = 0.03), greater symptoms of anxiety (r = -0.46; p < 0.01), more catastrophic thinking (r = -0.38; p = 0.03), and greater symptoms of depression (r = -0.42; p = 0.01). More time expressing surprise was associated with less health anxiety (r = -0.44; p < 0.01) and symptoms of depression (r = -0.52; p < 0.01). More time expressing fear was associated with less kinesiophobia (r = -0.35; p = 0.04). More time expressing disgust was associated with less catastrophic thinking (r = -0.37; p = 0.03) and less health anxiety (GAD-2; r = -0.42; p = 0.02) and symptoms of depression (r = -0.44; p < 0.01). There was no association between a clinicians' facial expression of emotions and patient experience with patient-clinician interactions.

CONCLUSION

The ability to measure a patient's mindset on the clinician's face confirms that clinicians are registering the psychological aspects of illness, whether they are consciously aware of them or not. Future research involving larger cohorts of patients, mapping clinician-patient interactions during consultation, and more sophisticated capture of nonverbal and verbal cues, including a broader range of emotional expressions, may help translate this innovation from the research setting to clinical practice.

CLINICAL RELEVANCE

Tools for measuring emotion through facial recognition could be used to train clinicians to become aware of the psychological aspects of health and to coach clinicians on effective communication strategies both for gentle reorientation of common misconceptions as well as for appropriate and timely diagnosis and treatment of psychological distress.

Psychological functioning in survivors of COVID-19: Evidence from recognition of fearful facial expressions

Evidence about the psychological functioning in individuals who survived the COVID-19 infectious is still rare in the literature. In this paper, we investigated fearful facial expressions recognition, as a behavioural means to assess psychological functioning. From May 15th, 2020 to January 30th, 2021, we enrolled sixty Italian individuals admitted in multiple Italian COVID-19 post-intensive care units. The detection and recognition of fearful facial expressions were assessed through an experimental task grounded on an attentional mechanism (i.e., the redundant target effect). According to the results, our participants showed an altered behaviour in detecting and recognizing fearful expressions. Specifically, their performance was in disagreement with the expected behavioural effect. Our study suggested altered processing of fearful expressions in individuals who survived the COVID-19 infectious. Such a difficulty might represent a crucial sign of psychological distress and it should be addressed in tailored psychological interventions in rehabilitative settings and after discharge.