Facial expression analysis with AFFDEX and FACET: A validation study

AI-enhanced analysis of naturalistic social interactions characterizes interaffective impairments in schizophrenia

Social deficits in schizophrenia have been attributed to an impaired attunement to mutual interaction, or "interaffectivity". While impairments in emotion recognition and facial expressivity in schizophrenia have been consistently reported, findings on mimicry and social synchrony are inconsistent, and previous studies have often lacked ecological validity. To investigate interaffective behavior in dyadic interactions in a real-world-like setting, 20 individuals with schizophrenia and 20 without mental disorder played a cooperative board game with a previously unacquainted healthy control participant. Facial expression analysis was conducted using Affectiva Emotion AI in iMotions 9.3. The contingency and state space distribution of emotional facial expressions was assessed using Mangold INTERACT. Psychotic symptoms, subjective stress, affectivity and game experience were evaluated through questionnaires. Due to a considerable between-group age difference, age-adjusted ANCOVA was performed. Overall, despite an unchanged subjective experience of the social interaction, individuals with schizophrenia exhibited reduced responsiveness to positive affective stimuli. Subjective game experience did not differ between groups. Descriptively, facial expressions in schizophrenia were generally more negative, with increased sadness and decreased joy. Facial mimicry was impaired specifically regarding joyful expressions in schizophrenia, which correlated with blunted affect as measured by the SANS. Dyadic interactions involving persons with schizophrenia were less attracted toward mutual joyful affective states. Only unadjusted for age, in the absence of emotional stimuli from their interaction partner, individuals with schizophrenia showed more angry and sad expressions. These impairments in interaffective processes may contribute to social dysfunction in schizophrenia and provide new avenues for future research.

Video analysis of interaction in schizophrenia reveals functionally relevant abnormalities

OBJECTIVE

Deficits of dyadic social interaction seem to diminish social functioning in schizophrenia. However, most previous studies are of a limited ecological validity due to decontextualized experimental conditions far off from real-world interaction. In this pilot study, we thus exposed participants to a more real-world-like situation to generate new hypotheses for research and therapeutic interventions.

METHODS

Dyads of either participants with schizophrenia (n = 21) or control participants without mental disorder (n = 21) were presented with a 5-min emotionally engaging movie. The subsequent uninstructed dyadic interaction was videotaped and analyzed by means of a semi-quantitative, software-supported behavioral analysis.

RESULTS

The patients with schizophrenia showed significant abnormalities regarding their social interaction, such as more negative verbalizations, a more open display of negative affect and gaze abnormalities. Their interaction behavior was mostly characterized by neutral affect, silence and avoidance of direct eye contact. Neutral affect was associated with poorer psychosocial performance. Verbal intelligence and empathy were associated with positive interaction variables, which were also not impaired by psychotic symptom severity.

CONCLUSION

In this real-world-like dyadic interaction, participants with schizophrenia show distinct abnormalities that are relevant to psychosocial performance and consistent with a hypothesized lack of attunement to interaffective situations.

Calibration trial of an innovative medical device (NEVVA© ) for the evaluation of pain in non-communicating patients in the intensive care unit

Background

Pain management is an essential and complex issue for non-communicative patients undergoing sedation in the intensive care unit (ICU). The Behavioral Pain Scale (BPS), although not perfect for assessing behavioral pain, is the gold standard based partly on clinical facial expression. NEVVA© , an automatic pain assessment tool based on facial expressions in critically ill patients, is a much-needed innovative medical device.

Methods

In this prospective pilot study, we recorded the facial expressions of critically ill patients in the medical ICU of Caen University Hospital using the iPhone and Smart Motion Tracking System (SMTS) software with the Facial Action Coding System (FACS) to measure human facial expressions metrically during sedation weaning. Analyses were recorded continuously, and BPS scores were collected hourly over two 8 h periods per day for 3 consecutive days. For this first stage, calibration of the innovative NEVVA© medical device algorithm was obtained by comparison with the reference pain scale (BPS).

Results

Thirty participants were enrolled between March and July 2022. To assess the acute severity of illness, the Sequential Organ Failure Assessment (SOFA) and the Simplified Acute Physiology Score (SAPS II) were recorded on ICU admission and were 9 and 47, respectively. All participants had deep sedation, assessed by a Richmond Agitation and Sedation scale (RASS) score of less than or equal to -4 at the time of inclusion. One thousand and six BPS recordings were obtained, and 130 recordings were retained for final calibration: 108 BPS recordings corresponding to the absence of pain and 22 BPS recordings corresponding to the presence of pain. Due to the small size of the dataset, a leave-one-subject-out cross-validation (LOSO-CV) strategy was performed, and the training results obtained the receiver operating characteristic (ROC) curve with an area under the curve (AUC) of 0.792. This model has a sensitivity of 81.8% and a specificity of 72.2%.

Conclusion

This pilot study calibrated the NEVVA© medical device and showed the feasibility of continuous facial expression analysis for pain monitoring in ICU patients. The next step will be to correlate this device with the BPS scale.

Association of intensity and dominance of CEOs' smiles with corporate performance

This study investigated whether the facial expressions of chief executive officers (CEOs) are associated with corporate performance. A photograph of the CEO or president of each company that appeared on the Fortune Global 500 list for 2018 was taken from the company's official website. The smile intensity and action unit activation in each face were calculated using a pre-trained machine learning algorithm, FACET. The results revealed a positive association between smile intensity and company profit, even when controlling for the company's geographic location (Western culture versus others) and the CEO's gender. Furthermore, when the type of smile was examined with the activation of each action unit, this significant positive association was identified in the dominant smile but not in the reward and affiliative smiles. Relationships among the leader's smile intensity, group strategy, and group performance are discussed.

Objective Measurement of Social Gaze and Smile Behaviors in Children with Suspected Autism Spectrum Disorder During Administration of the Autism Diagnostic Observation Schedule, 2nd Edition

Best practice for the assessment of autism spectrum disorder (ASD) symptom severity relies on clinician ratings of the Autism Diagnostic Observation Schedule, 2nd Edition (ADOS-2), but the association of these ratings with objective measures of children's social gaze and smiling is unknown. Sixty-six preschool-age children (49 boys, M = 39.97 months, SD = 10.58) with suspected ASD (61 confirmed ASD) were administered the ADOS-2 and provided social affect calibrated severity scores (SA CSS). Children's social gaze and smiling during the ADOS-2, captured with a camera contained in eyeglasses worn by the examiner and parent, were obtained via a computer vision processing pipeline. Children who gazed more at their parents (p = .04) and whose gaze at their parents involved more smiling (p = .02) received lower social affect severity scores, indicating fewer social affect symptoms, adjusted R2 = .15, p = .003.

Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders

Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.

Evaluation of a motorised patient transfer device based on perceived workload, technology acceptance, and emotional states

PURPOSE

The high prevalence of musculoskeletal disorders (MSDs) among healthcare workers is partly attributed to the low adoption of patient transfer assistive devices. This study aimed to evaluate the nurses' perceived workload, technology acceptance, and emotional states during the use of the sliding board (SB) and mechanical intervention in the form of a Motorised Patient Transfer Device (MPTD).

METHODS

The SB and MPTD activities were performed by seven nurses on a simulated patient. The nurses' facial expressions were recorded during the trial. The NASA Task Load Index and technology acceptance questionnaire were also assessed.

RESULTS

The MPTD significantly reduced the mean overall NASA-TLX score by 68.7% (p = 0.004) and increased the overall acceptance score (median = 8.30) by 21.2% (p = 0.016) when compared to the SB (median = 6.85). All the subjects reported positive feelings towards MPTD. However, facial expression analysis showed that the nurses had a significantly higher peak density of fear while using MPTD (p = 0.016). Besides, there was no improvement in the negative valence and contempt emotion compared to the SB.

CONCLUSION

Overall, nurses showed positive perceptions and acceptance of MPTD even when they experienced negative emotions.IMPLICATIONS FOR REHABILITATIONThe Motorised Patient Transfer Device (MPTD) reduced the perceived workload of nurses and showed a higher acceptance level compared to the commonly used baseline device (SB).Factors that attributed to the nurses' negative emotions can be used to improve technology and patient transfer processes.More training should be given to familiarise the health practitioners with the new assistive device to reduce their fear of technology.

The relationship between charitable giving and emotional facial expressions: Results from affective computing

This study investigated the relationship between emotional states (valence, arousal, and six basic emotions) and donation size in pet charities, and it compared the effectiveness of affective computing and emotion self-report methods in assessing attractiveness. Using FaceReader software and self-report, we measured the emotional states of participants (N = 45) during the donation task. The results showed that sadness, happiness, and anger were significantly related to donation size. Sadness and anger increased donations, whereas happiness decreased them. Arousal was not significantly correlated with the willingness to donate. These results are supported by both methods, whereas the self-reported data regarding the association of surprise, fear, and disgust with donation size are inconclusive. Thus, unpleasant emotions increase donation size, and combining affective computing with self-reported data improves the prediction of the effectiveness of a charity appeal. This study contributes to the understanding of the relationship between emotions and charitable behavior toward pet charities and evaluates the effectiveness of marketing mix elements using affective computing. The limitations include the laboratory setting for this experiment and the lack of measurement of prolonged and repeated exposure to unpleasant charity appeals.

Measuring facial mimicry: Affdex vs. EMG

Facial mimicry is the automatic imitation of the facial affect expressions of others. It serves as an important component of interpersonal communication and affective co-experience. Facial mimicry has so far been measured by Electromyography (EMG), which requires a complex measuring apparatus. Recently, software for measuring facial expressions have become available, but it is still unclear how well it is suited for measuring facial mimicry. This study investigates the comparability of the automated facial coding software Affdex with EMG for measuring facial mimicry. For this purpose, facial mimicry was induced in 33 subjects by presenting naturalistic affect-expressive video sequences (anger, joy). The response of the subjects is measured simultaneously by facial EMG (corrugator supercilii muscle, zygomaticus major muscle) and by Affdex (action units lip corner puller and brow lowerer and affects joy and anger). Subsequently, the correlations between the measurement results of EMG and Affdex were calculated. After the presentation of the joy stimulus, there was an increase in zygomaticus muscle activity (EMG) about 400 ms after stimulus onset and an increase in joy and lip corner puller activity (Affdex) about 1200 ms after stimulus onset. The joy and the lip corner puller activity detected by Affdex correlate significantly with the EMG activity. After presentation of the anger stimulus, corrugator muscle activity (EMG) also increased approximately 400 ms after stimulus onset, whereas anger and brow lowerer activity (Affdex) showed no response. During the entire measurement interval, anger activity and brow lowerer activity (Affdex) did not correlate with corrugator muscle activity (EMG). Using Affdex, the facial mimicry response to a joy stimulus can be measured, but it is detected approximately 800 ms later compared to the EMG. Thus, electromyography remains the tool of choice for studying subtle mimic processes like facial mimicry.

Development of a Framework for the Communication System Based on KNX for an Interactive Space for UX Evaluation

Domotics (Home Automation) aims to improve the quality of life of people by integrating intelligent systems within inhabitable spaces. While traditionally associated with smart home systems, these technologies have potential for User Experience (UX) research. By emulating environments to test products and services, and integrating non-invasive user monitoring tools for emotion recognition, an objective UX evaluation can be performed. To achieve this objective, a testing booth was built and instrumented with devices based on KNX, an international standard for home automation, to conduct experiments and ensure replicability. A framework was designed based on Python to synchronize KNX systems with emotion recognition tools; the synchronization of these data allows finding patterns during the interaction process. To evaluate this framework, an experiment was conducted in a simulated laundry room within the testing booth to analyze the emotional responses of participants while interacting with prototypes of new detergent bottles. Emotional responses were contrasted with traditional questionnaires to determine the viability of using non-invasive methods. Using emulated environments alongside non-invasive monitoring tools allowed an immersive experience for participants. These results indicated that the testing booth can be implemented for a robust UX evaluation methodology.

Py-Feat: Python Facial Expression Analysis Toolbox

Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state-of-the-art models require considerable domain expertise that is not traditionally incorporated into social science training programs. Furthermore, there is a notable absence of user-friendly and open-source software that provides a comprehensive set of tools and functions that support facial expression research. In this paper, we introduce Py-Feat, an open-source Python toolbox that provides support for detecting, preprocessing, analyzing, and visualizing facial expression data. Py-Feat makes it easy for domain experts to disseminate and benchmark computer vision models and also for end users to quickly process, analyze, and visualize face expression data. We hope this platform will facilitate increased use of facial expression data in human behavior research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-023-00191-4.

Assessment of the human response to acute mental stress-An overview and a multimodal study

Numerous vital signs are reported in association with stress response assessment, but their application varies widely. This work provides an overview over methods for stress induction and strain assessment, and presents a multimodal experimental study to identify the most important vital signs for effective assessment of the response to acute mental stress. We induced acute mental stress in 65 healthy participants with the Mannheim Multicomponent Stress Test and acquired self-assessment measures (Likert scale, Self-Assessment Manikin), salivary α-amylase and cortisol concentrations as well as 60 vital signs from biosignals, such as heart rate variability parameters, QT variability parameters, skin conductance level, and breath rate. By means of statistical testing and a self-optimizing logistic regression, we identified the most important biosignal vital signs. Fifteen biosignal vital signs related to ventricular repolarization variability, blood pressure, skin conductance, and respiration showed significant results. The logistic regression converged with QT variability index, left ventricular work index, earlobe pulse arrival time, skin conductance level, rise time and number of skin conductance responses, breath rate, and breath rate variability (F1 = 0.82). Self-assessment measures indicated successful stress induction. α-amylase and cortisol showed effect sizes of -0.78 and 0.55, respectively. In summary, the hypothalamic-pituitary-adrenocortical axis and sympathetic nervous system were successfully activated. Our findings facilitate a coherent and integrative understanding of the assessment of the stress response and help to align applications and future research concerning acute mental stress.

The Emerging Role of Artificial Intelligence Tools for Outcome Measurement in Facial Reanimation Surgery: A Review

Importance: Surgeons treating facial paralysis with reanimation surgery measure the outcomes of surgery and adjust treatment to each patient's needs. Our objective is to review the current subjective facial paralysis assessment tools and the emerging computer-based objective analysis, which may involve artificial intelligence. Observations: In recent years, many new automated approaches to outcome measurement in facial reanimation surgery have been developed. Most of these tools utilize artificial intelligence to analyze emotional expression and symmetry of facial landmarks. Other tools have provided automated approaches to existing clinician-guided scales. Conclusions: Newly developed computer-based tools using artificial intelligence have been developed to both improve existing clinician-graded scales and provide new approaches to facial symmetry and emotional expressivity analysis.

Synchronized affect in shared experiences strengthens social connection

People structure their days to experience events with others. We gather to eat meals, watch TV, and attend concerts together. What constitutes a shared experience and how does it manifest in dyadic behavior? The present study investigates how shared experiences-measured through emotional, motoric, physiological, and cognitive alignment-promote social bonding. We recorded the facial expressions and electrodermal activity (EDA) of participants as they watched four episodes of a TV show for a total of 4 h with another participant. Participants displayed temporally synchronized and spatially aligned emotional facial expressions and the degree of synchronization predicted the self-reported social connection ratings between viewing partners. We observed a similar pattern of results for dyadic physiological synchrony measured via EDA and their cognitive impressions of the characters. All four of these factors, temporal synchrony of positive facial expressions, spatial alignment of expressions, EDA synchrony, and character impression similarity, contributed to a latent factor of a shared experience that predicted social connection. Our findings suggest that the development of interpersonal affiliations in shared experiences emerges from shared affective experiences comprising synchronous processes and demonstrate that these complex interpersonal processes can be studied in a holistic and multi-modal framework leveraging naturalistic experimental designs.

Development of a Universal Validation Protocol and an Open-Source Database for Multi-Contextual Facial Expression Recognition

Facial expression recognition (FER) poses a complex challenge due to diverse factors such as facial morphology variations, lighting conditions, and cultural nuances in emotion representation. To address these hurdles, specific FER algorithms leverage advanced data analysis for inferring emotional states from facial expressions. In this study, we introduce a universal validation methodology assessing any FER algorithm's performance through a web application where subjects respond to emotive images. We present the labelled data database, FeelPix, generated from facial landmark coordinates during FER algorithm validation. FeelPix is available to train and test generic FER algorithms, accurately identifying users' facial expressions. A testing algorithm classifies emotions based on FeelPix data, ensuring its reliability. Designed as a computationally lightweight solution, it finds applications in online systems. Our contribution improves facial expression recognition, enabling the identification and interpretation of emotions associated with facial expressions, offering profound insights into individuals' emotional reactions. This contribution has implications for healthcare, security, human-computer interaction, and entertainment.

Human and machine recognition of dynamic and static facial expressions: prototypicality, ambiguity, and complexity

A growing body of research suggests that movement aids facial expression recognition. However, less is known about the conditions under which the dynamic advantage occurs. The aim of this research was to test emotion recognition in static and dynamic facial expressions, thereby exploring the role of three featural parameters (prototypicality, ambiguity, and complexity) in human and machine analysis. In two studies, facial expression videos and corresponding images depicting the peak of the target and non-target emotion were presented to human observers and the machine classifier (FACET). Results revealed higher recognition rates for dynamic stimuli compared to non-target images. Such benefit disappeared in the context of target-emotion images which were similarly well (or even better) recognised than videos, and more prototypical, less ambiguous, and more complex in appearance than non-target images. While prototypicality and ambiguity exerted more predictive power in machine performance, complexity was more indicative of human emotion recognition. Interestingly, recognition performance by the machine was found to be superior to humans for both target and non-target images. Together, the findings point towards a compensatory role of dynamic information, particularly when static-based stimuli lack relevant features of the target emotion. Implications for research using automatic facial expression analysis (AFEA) are discussed.

A Critique of Automated Approaches to Code Facial Expressions: What Do Researchers Need to Know?

Facial expression recognition software is becoming more commonly used by affective scientists to measure facial expressions. Although the use of this software has exciting implications, there are persistent and concerning issues regarding the validity and reliability of these programs. In this paper, we highlight three of these issues: biases of the programs against certain skin colors and genders; the common inability of these programs to capture facial expressions made in non-idealized conditions (e.g., "in the wild"); and programs being forced to adopt the underlying assumptions of the specific theory of emotion on which each software is based. We then discuss three directions for the future of affective science in the area of automated facial coding. First, researchers need to be cognizant of exactly how and on which data sets the machine learning algorithms underlying these programs are being trained. In addition, there are several ethical considerations, such as privacy and data storage, surrounding the use of facial expression recognition programs. Finally, researchers should consider collecting additional emotion data, such as body language, and combine these data with facial expression data in order to achieve a more comprehensive picture of complex human emotions. Facial expression recognition programs are an excellent method of collecting facial expression data, but affective scientists should ensure that they recognize the limitations and ethical implications of these programs.

Designing a neuroclinical assessment of empathy deficits in psychopathy based on the Zipper Model of Empathy

Ho, M.H., Kemp, B.T., Eisenbarth, H. & Rijnders, R.J.P. Designing a neuroclinical assessment of empathy deficits in psychopathy based on the Zipper Model of Empathy. NEUROSCI BIOBEHAV REV YY(Y) XXX-XXX, 2023. The heterogeneity of the literature on empathy highlights its multidimensional and dynamic nature and affects unclear descriptions of empathy in the context of psychopathology. The Zipper Model of Empathy integrates current theories of empathy and proposes that empathy maturity is dependent on whether contextual and personal factors push affective and cognitive processes together or apart. This concept paper therefore proposes a comprehensive battery of physiological and behavioral measures to empirically assess empathy processing according to this model with an application for psychopathic personality. We propose using the following measures to assess each component of this model: (1) facial electromyography; (2) the Emotion Recognition Task; (3) the Empathy Accuracy task and physiological measures (e.g., heart rate); (4) a selection of Theory of Mind tasks and an adapted Dot Perspective Task, and; (5) an adjusted Charity Task. Ultimately, we hope this paper serves as a starting point for discussion and debate on defining and assessing empathy processing, to encourage research to falsify and update this model to improve our understanding of empathy.

Subjective experience, self-efficacy, and motivation of professional football referees during the COVID-19 pandemic

The present multi-study article investigates the subjective experience of professional football (a.k.a. soccer) referees and players during the COVID-19 pandemic and the so-called ghost games (i.e., games without supporters). Referees from the Austrian Football Association completed questionnaires inquiring about self-efficacy, motivation, and general personal observations and perceptions (e.g., arousal or confidence). In addition, two players and one referee in the Austrian Football Bundesliga were interviewed retrospectively regarding their subjective experience during ghost games and the effects of emotions on behavior and performance using semi-structured, video-taped interviews. Results of the referee survey indicate that the most profound differences between regular games and ghost games lie in the domain of intrinsic motivation and multiple aspects of subjective experience. Specifically, the experience in ghost games compared with regular games was reported by referees as being significantly less motivating, less excited/tense, less emotional, less focused, and overall, more negative, despite being easier to referee and the players behaving more positively. Qualitative analyses of the video-taped interview footage indicated (i) substantial inter-individual variability regarding the extent of the effect of the empty stadiums on the subjective experience of emotions, (ii) consequently, different strategies to regulate emotions and arousal from suboptimal to optimal levels, both before and during competition, and (iii) interactions between reported emotions, arousal, motivation, self-confidence, behavior and performance on the pitch. In addition, non-verbal expressions of emotion were captured using fully automated AI-software that coded facial movements during interviews. The results of this exploratory facial expression analysis revealed varying degrees of arousal and valence in relation to the content of the statements during the interviews, demonstrating the convergent validity of our findings. Our findings contribute to the growing literature on the effects of football games without fans during the COVID-19 pandemic and provide insights into the subjective experience of professional football referees. Concerning referees and players alike, emotions are investigated as potential processes related to home-field advantage and performance in professional football by means of a multi-methods approach. Further, the combination of qualitative and quantitative measures-as well as verbal and non-verbal communication channels-can deepen our understanding of the emotional influence of (missing) spectators on the subjective experience and the behavior of sports professionals is discussed.

Facial Expressions as an Index of Listening Difficulty and Emotional Response

Knowledge about listening difficulty experienced during a task can be used to better understand speech perception processes, to guide amplification outcomes, and can be used by individuals to decide whether to participate in communication. Another factor affecting these decisions is individuals' emotional response which has not been measured objectively previously. In this study, we describe a novel method of measuring listening difficulty and affect of individuals in adverse listening situations using automatic facial expression algorithm. The purpose of our study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We recorded speech recognition scores, facial expressions, subjective listening effort scores, and subjective emotional responses in 33 young participants with normal hearing. We used the signal-to-noise ratios of -1, +2, and +5 dB SNR and quiet conditions to vary the difficulty level. We found that facial expression of confusion and frustration increased with increase in difficulty level, but not with change in each level. We also found a relationship between facial expressions and both subjective emotion ratings and subjective listening effort. Emotional responses in the form of facial expressions show promise as a measure of affect and listening difficulty. Further research is needed to determine the specific contribution of affect to communication in challenging listening environments.

Lighting environmental assessment in enclosed spaces based on emotional model

Lighting assessment in special operating environments like enclosed spaces is of great research significance and value. In addition to investigating the visual ergonomics of workers, the emotional monitoring and guidance of workers in enclosed spaces is also a research focus. Based on the Circumplex emotion model theory, this paper designs an experiment to assess emotions in an enclosed space with 6 different lighting settings (2 correlated color temperature (CCT) × 3 illuminance levels). From the perspective of subjective assessment, participants used a rapid sensory analysis method (Check-all-that-apply, CATA) and a Subjective Coordinate Scale (SCS) method for rapid ambience perception checking and emotional self-reporting of lighting settings. From the perspective of objective evaluation, the participants' facial expressions were recorded during the experiment using a camera, and then the recorded facial expressions were automatically analyzed and predicted using FaceReader (FRE) software. The CATA and SCS showed similar results, with the 3100 K × 600 lx, 3100 K × 1000 lx and 6500 K × 600 lx lighting settings creating a relaxed, pleasant emotion in participants, the 6500 K × 1000 lx setting creating an excited, tense atmosphere, and the low illumination level settings of 3100 K × 250 lx and 6500 K × 250 lx made participants feel tired and frustrated. The results of the objective emotion analysis indicate that the FRE was able to effectively identify differences in participants' emotions in response to different lighting settings and was consistent with the results of participants' subjective emotion reports. This laboratory study validates that the three methods can effectively assess the enclosed space lighting settings, and provides a reference for further research on the enclosed space lighting settings and emotional monitoring of workers in the future.

Emotional Valence from Facial Expression as an Experience Audit Tool: An Empirical Study in the Context of Opera Performance

This paper aims to explore the potential offered by emotion recognition systems to provide a feasible response to the growing need for audience understanding and development in the field of arts organizations. Through an empirical study, it was investigated whether the emotional valence measured on the audience through an emotion recognition system based on facial expression analysis can be used with an experience audit to: (1) support the understanding of the emotional responses of customers toward any clue that characterizes a staged performance; and (2) systematically investigate the customer's overall experience in terms of their overall satisfaction. The study was carried out in the context of opera live shows in the open-air neoclassical theater Arena Sferisterio in Macerata, during 11 opera performances. A total of 132 spectators were involved. Both the emotional valence provided by the considered emotion recognition system and the quantitative data related to customers' satisfaction, collected through a survey, were considered. Results suggest how collected data can be useful for the artistic director to estimate the audience's overall level of satisfaction and make choices about the specific characteristics of the performance, and that emotional valence measured on the audience during the show can be useful to predict overall customer satisfaction, as measured using traditional self-report methods.

Malocclusion severity and smile features: Is there an association?

INTRODUCTION

This observational study investigated the relationship between malocclusion and smiling.

METHODS

Adolescents and young adults (n = 72; aged 16-25 years) were identified according to their Dental Aesthetic Index (DAI) and allocated to 3 groups: (1) malocclusion group (n = 24; DAI ≥31), (2) retention group (n = 24; pretreatment DAI ≥31) with a prior malocclusion that had been corrected by orthodontic treatment, (3) control group with no-to-minor malocclusion (n = 24; DAI ≤25). Participants were requested to watch an amusing video. Based on the Facial Action Coding System, automated pattern recognition was used to detect smile episodes and assess their frequency, duration, genuineness, intensity, and extent of tooth show. Demographics, Big Five personality dimensions, and self-perceived smile esthetics-related quality of life were collected from all participants via questionnaires. Data were analyzed by mixed-model analysis and adjusted for possible confounders.

RESULTS

Patients from the malocclusion and retention groups smiled significantly less than participants from the control group, with the duration of smiles and smiling time being around half those of control subjects. Smile genuineness, smile intensity, and teeth shown did not differ across groups. Personality traits did not differ significantly among the 3 groups, whereas the malocclusion group scored around 30% less for dental self-confidence than the other 2 groups.

CONCLUSIONS

Patients with severe malocclusion tend to smile less, but the features of their smiles are similar to those without malocclusion. A lower propensity to smile in patients with a corrected malocclusion may persist after orthodontic treatment.

Training machine learning algorithms for automatic facial coding: The role of emotional facial expressions' prototypicality

Automatic facial coding (AFC) is a promising new research tool to efficiently analyze emotional facial expressions. AFC is based on machine learning procedures to infer emotion categorization from facial movements (i.e., Action Units). State-of-the-art AFC accurately classifies intense and prototypical facial expressions, whereas it is less accurate for non-prototypical and less intense facial expressions. A potential reason might be that AFC is typically trained with standardized and prototypical facial expression inventories. Because AFC would be useful to analyze less prototypical research material as well, we set out to determine the role of prototypicality in the training material. We trained established machine learning algorithms either with standardized expressions from widely used research inventories or with unstandardized emotional facial expressions obtained in a typical laboratory setting and tested them on identical or cross-over material. All machine learning models' accuracies were comparable when trained and tested with held-out dataset from the same dataset (acc. = [83.4% to 92.5%]). Strikingly, we found a substantial drop in accuracies for models trained with the highly prototypical standardized dataset when tested in the unstandardized dataset (acc. = [52.8%; 69.8%]). However, when they were trained with unstandardized expressions and tested with standardized datasets, accuracies held up (acc. = [82.7%; 92.5%]). These findings demonstrate a strong impact of the training material's prototypicality on AFC's ability to classify emotional faces. Because AFC would be useful for analyzing emotional facial expressions in research or even naturalistic scenarios, future developments should include more naturalistic facial expressions for training. This approach will improve the generalizability of AFC to encode more naturalistic facial expressions and increase robustness for future applications of this promising technology.

Clients' Facial Expressions of Self-Compassion, Self-Criticism, and Self-Protection in Emotion-Focused Therapy Videos

Clients' facial expressions allow psychotherapists to gather more information about clients' emotional processing. This study aims to examine and investigate the facial Action Units (AUs) of self-compassion, self-criticism, and self-protection within real Emotion-Focused Therapy (EFT) sessions. For this purpose, we used the facial analysis software iMotions. Twelve video sessions were selected for the analysis based on specific criteria. For self-compassion, the following AUs were significant: AUs 4 (brow furrow), 15 (lip corner depressor), and the AU12_smile (lip corner puller). For self-criticism, iMotions identified the AUs 2 (outer brow raise), AU1 (inner brow raise), AU7 (lid tighten), AU12_smirk (unilateral lip corner puller), and AU43 (eye closure). Self-protection was combined using the occurrence of AUs 1 and 4 and AU12_smirk. Moreover, the findings support the significance of discerning self-compassion and self-protection as two different concepts.

Automatic facial coding predicts self-report of emotion, advertisement and brand effects elicited by video commercials

Introduction

Consumers' emotional responses are the prime target for marketing commercials. Facial expressions provide information about a person's emotional state and technological advances have enabled machines to automatically decode them.

Method

With automatic facial coding we investigated the relationships between facial movements (i.e., action unit activity) and self-report of commercials advertisement emotion, advertisement and brand effects. Therefore, we recorded and analyzed the facial responses of 219 participants while they watched a broad array of video commercials.

Results

Facial expressions significantly predicted self-report of emotion as well as advertisement and brand effects. Interestingly, facial expressions had incremental value beyond self-report of emotion in the prediction of advertisement and brand effects. Hence, automatic facial coding appears to be useful as a non-verbal quantification of advertisement effects beyond self-report.

Discussion

This is the first study to measure a broad spectrum of automatically scored facial responses to video commercials. Automatic facial coding is a promising non-invasive and non-verbal method to measure emotional responses in marketing.

Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration

In animal research, automation of affective states recognition has so far mainly addressed pain in a few species. Emotional states remain uncharted territories, especially in dogs, due to the complexity of their facial morphology and expressions. This study contributes to fill this gap in two aspects. First, it is the first to address dog emotional states using a dataset obtained in a controlled experimental setting, including videos from (n = 29) Labrador Retrievers assumed to be in two experimentally induced emotional states: negative (frustration) and positive (anticipation). The dogs' facial expressions were measured using the Dogs Facial Action Coding System (DogFACS). Two different approaches are compared in relation to our aim: (1) a DogFACS-based approach with a two-step pipeline consisting of (i) a DogFACS variable detector and (ii) a positive/negative state Decision Tree classifier; (2) An approach using deep learning techniques with no intermediate representation. The approaches reach accuracy of above 71% and 89%, respectively, with the deep learning approach performing better. Secondly, this study is also the first to study explainability of AI models in the context of emotion in animals. The DogFACS-based approach provides decision trees, that is a mathematical representation which reflects previous findings by human experts in relation to certain facial expressions (DogFACS variables) being correlates of specific emotional states. The deep learning approach offers a different, visual form of explainability in the form of heatmaps reflecting regions of focus of the network's attention, which in some cases show focus clearly related to the nature of particular DogFACS variables. These heatmaps may hold the key to novel insights on the sensitivity of the network to nuanced pixel patterns reflecting information invisible to the human eye.

Discrete emotions discovered by contactless measurement of facial blood flows

ABSTRACTExperiential and behavioural aspects of emotions can be measured readily but developing a contactless measure of emotions' physiological aspects has been a major challenge. We hypothesised that different emotion-evoking films can produce distinctive facial blood flow patterns that can serve as physiological signatures of discrete emotions. To test this hypothesis, we created a new Transdermal Optical Imaging system that uses a conventional video camera to capture facial blood flows in a contactless manner. Using this and deep machine learning, we analysed videos of the faces of people as they viewed film clips that elicited joy, sadness, disgust, fear or a neutral state. We found that each of these elicited a distinct blood flow pattern in the facial epidermis, and that Transdermal Optical Imaging is an effective contactless and inexpensive tool to the reveal physiological correlates of discrete emotions.

Do Deepfakes Adequately Display Emotions? A Study on Deepfake Facial Emotion Expression

Recent technological advancements in Artificial Intelligence make it easy to create deepfakes and hyper-realistic videos, in which images and video clips are processed to create fake videos that appear authentic. Many of them are based on swapping faces without the consent of the person whose appearance and voice are used. As emotions are inherent in human communication, studying how deepfakes transfer emotional expressions from original to fakes is relevant. In this work, we conduct an in-depth study on facial emotional expression in deepfakes using a well-known face swap-based deepfake database. Firstly, we extracted the photograms from their videos. Then, we analyzed the emotional expression in the original and faked versions of video recordings for all performers in the database. Results show that emotional expressions are not adequately transferred between original recordings and the deepfakes created from them. High variability in emotions and performers detected between original and fake recordings indicates that performer emotion expressiveness should be considered for better deepfake generation or detection.

Identifying and describing subtypes of spontaneous empathic facial expression production in autistic adults

BACKGROUND

It is unclear whether atypical patterns of facial expression production metrics in autism reflect the dynamic and nuanced nature of facial expressions across people or a true diagnostic difference. Furthermore, the heterogeneity observed across autism symptomatology suggests a need for more adaptive and personalized social skills programs. Towards this goal, it would be useful to have a more concrete and empirical understanding of the different expressiveness profiles within the autistic population and how they differ from neurotypicals.

METHODS

We used automated facial coding and an unsupervised clustering approach to limit inter-individual variability in facial expression production that may have otherwise obscured group differences in previous studies, allowing an "apples-to-apples" comparison between autistic and neurotypical adults. Specifically, we applied k-means clustering to identify subtypes of facial expressiveness in an autism group (N = 27) and a neurotypical control group (N = 57) separately. The two most stable clusters from these analyses were then further characterized and compared based on their expressiveness and emotive congruence to emotionally charged stimuli.

RESULTS

Our main finding was that a subset of autistic adults in our sample show heightened spontaneous facial expressions irrespective of image valence. We did not find evidence for greater incongruous (i.e., inappropriate) facial expressions in autism. Finally, we found a negative trend between expressiveness and emotion recognition within the autism group.

CONCLUSION

The results from our previous study on self-reported empathy and current expressivity findings point to a higher degree of facial expressions recruited for emotional resonance in autism that may not always be adaptive (e.g., experiencing similar emotional resonance regardless of valence). These findings also build on previous work indicating that facial expression intensity is not diminished in autism and suggest the need for intervention programs to focus on emotion recognition and social skills in the context of both negative and positive emotions.

Test–Retest Reliability in Automated Emotional Facial Expression Analysis: Exploring FaceReader 8.0 on Data from Typically Developing Children and Children with Autism

Automated emotional facial expression analysis (AEFEA) is used widely in applied research, including the development of screening/diagnostic systems for atypical human neurodevelopmental conditions. The validity of AEFEA systems has been systematically studied, but their test–retest reliability has not been researched thus far. We explored the test–retest reliability of a specific AEFEA software, Noldus FaceReader 8.0 (FR8; by Noldus Information Technology). We collected intensity estimates for 8 repeated emotions through FR8 from facial video recordings of 60 children: 31 typically developing children and 29 children with autism spectrum disorder. Test–retest reliability was imperfect in 20% of cases, affecting a substantial proportion of data points; however, the test–retest differences were small. This shows that the test–retest reliability of FR8 is high but not perfect. A proportion of cases which initially failed to show perfect test–retest reliability reached it in a subsequent analysis by FR8. This suggests that repeated analyses by FR8 can, in some cases, lead to the “stabilization” of emotion intensity datasets. Under ANOVA, the test–retest differences did not influence the pattern of cross-emotion and cross-group effects and interactions. Our study does not question the validity of previous results gained by AEFEA technology, but it shows that further exploration of the test–retest reliability of AEFEA systems is desirable.

Investigating the Relationship between Facial Mimicry and Empathy

Facial expressions play a key role in interpersonal communication when it comes to negotiating our emotions and intentions, as well as interpreting those of others. Research has shown that we can connect to other people better when we exhibit signs of empathy and facial mimicry. However, the relationship between empathy and facial mimicry is still debated. Among the factors contributing to the difference in results across existing studies is the use of different instruments for measuring both empathy and facial mimicry, as well as often ignoring the differences across various demographic groups. This study first looks at the differences in the empathetic abilities of people across different demographic groups based on gender, ethnicity and age. The empathetic ability is measured based on the Empathy Quotient, capturing a balanced representation of both emotional and cognitive empathy. Using statistical and machine learning methods, this study then investigates the correlation between the empathetic ability and facial mimicry of subjects in response to images portraying different emotions displayed on a computer screen. Unlike the existing studies measuring facial mimicry using electromyography, this study employs a technology detecting facial expressions based on video capture and deep learning. This choice was made in the context of increased online communication during and after the COVID-19 pandemic. The results of this study confirm the previously reported difference in the empathetic ability between females and males. However, no significant difference in empathetic ability was found across different age and ethnic groups. Furthermore, no strong correlation was found between empathy and facial reactions to faces portraying different emotions shown on a computer screen. Overall, the results of this study can be used to inform the design of online communication technologies and tools for training empathy team leaders, educators, social and healthcare providers.

The Current View on the Paradox of Pain in Autism Spectrum Disorders

Autism spectrum disorder (ASD) is a neurodevelopmental disorder, which affects 1 in 44 children and may cause severe disabilities. Besides socio-communicational difficulties and repetitive behaviors, ASD also presents as atypical sensorimotor function and pain reactivity. While chronic pain is a frequent co-morbidity in autism, pain management in this population is often insufficient because of difficulties in pain evaluation, worsening their prognosis and perhaps driving higher mortality rates. Previous observations have tended to oversimplify the experience of pain in autism as being insensitive to painful stimuli. Various findings in the past 15 years have challenged and complicated this dogma. However, a relatively small number of studies investigates the physiological correlates of pain reactivity in ASD. We explore the possibility that atypical pain perception in people with ASD is mediated by alterations in pain perception, transmission, expression and modulation, and through interactions between these processes. These complex interactions may account for the great variability and sometimes contradictory findings from the studies. A growing body of evidence is challenging the idea of alterations in pain processing in ASD due to a single factor, and calls for an integrative view. We propose a model of the pain cycle that includes the interplay between the molecular and neurophysiological pathways of pain processing and it conscious appraisal that may interfere with pain reactivity and coping in autism. The role of social factors in pain-induced response is also discussed. Pain assessment in clinical care is mostly based on subjective rather than objective measures. This review clarifies the strong need for a consistent methodology, and describes innovative tools to cope with the heterogeneity of pain expression in ASD, enabling individualized assessment. Multiple measures, including self-reporting, informant reporting, clinician-assessed, and purely physiological metrics may provide more consistent results. An integrative view on the regulation of the pain cycle offers a more robust framework to characterize the experience of pain in autism.

Automated recognition of pain in cats

Facial expressions in non-human animals are closely linked to their internal affective states, with the majority of empirical work focusing on facial shape changes associated with pain. However, existing tools for facial expression analysis are prone to human subjectivity and bias, and in many cases also require special expertise and training. This paper presents the first comparative study of two different paths towards automatizing pain recognition in facial images of domestic short haired cats (n = 29), captured during ovariohysterectomy at different time points corresponding to varying intensities of pain. One approach is based on convolutional neural networks (ResNet50), while the other-on machine learning models based on geometric landmarks analysis inspired by species specific Facial Action Coding Systems (i.e. catFACS). Both types of approaches reach comparable accuracy of above 72%, indicating their potential usefulness as a basis for automating cat pain detection from images.

Measuring the Candidates' Emotions in Political Debates Based on Facial Expression Recognition Techniques

This article presents the analysis of the main Spanish political candidates for the elections to be held on April 2019. The analysis focuses on the Facial Expression Analysis (FEA), a technique widely used in neuromarketing research. It allows to identify the micro-expressions that are very brief, involuntary. They are signals of hidden emotions that cannot be controlled voluntarily. The video with the final interventions of every candidate has been post-processed using the classification algorithms given by the iMotions's AFFDEX platform. We have then analyzed these data. Firstly, we have identified and compare the basic emotions showed by each politician. Second, we have associated the basic emotions with specific moments of the candidate's speech, identifying the topics they address and relating them directly to the expressed emotion. Third, we have analyzed whether the differences shown by each candidate in every emotion are statistically significant. In this sense, we have applied the non-parametric chi-squared goodness-of-fit test. We have also considered the ANOVA analysis in order to test whether, on average, there are differences between the candidates. Finally, we have checked if there is consistency between the results provided by different surveys from the main media in Spain regarding the evaluation of the debate and those obtained in our empirical analysis. A predominance of negative emotions has been observed. Some inconsistencies were found between the emotion expressed in the facial expression and the verbal content of the message. Also, evidences got from statistical analysis confirm that the differences observed between the various candidates with respect to the basic emotions, on average, are statistically significant. In this sense, this article provides a methodological contribution to the analysis of the public figures' communication, which could help politicians to improve the effectiveness of their messages identifying and evaluating the intensity of the expressed emotions.

Affective Response Categories—Toward Personalized Reactions in Affect-Adaptive Tutoring Systems

Affect-adaptive tutoring systems detect the current emotional state of the learner and are capable of adequately responding by adapting the learning experience. Adaptations could be employed to manipulate the emotional state in a direction favorable to the learning process; for example, contextual help can be offered to mitigate frustration, or lesson plans can be accelerated to avoid boredom. Safety-critical situations, in which wrong decisions and behaviors can have fatal consequences, may particularly benefit from affect-adaptive tutoring systems, because accounting for affecting responses during training may help develop coping strategies and improve resilience. Effective adaptation, however, can only be accomplished when knowing which emotions benefit high learning performance in such systems. The results of preliminary studies indicate interindividual differences in the relationship between emotion and performance that require consideration by an affect-adaptive system. To that end, this article introduces the concept of Affective Response Categories (ARCs) that can be used to categorize learners based on their emotion-performance relationship. In an experimental study, N = 50 subjects (33% female, 19–57 years, M = 32.75, SD = 9.8) performed a simulated airspace surveillance task. Emotional valence was detected using facial expression analysis, and pupil diameters were used to indicate emotional arousal. A cluster analysis was performed to group subjects into ARCs based on their individual correlations of valence and performance as well as arousal and performance. Three different clusters were identified, one of which showed no correlations between emotion and performance. The performance of subjects in the other two clusters benefitted from negative arousal and differed only in the valence-performance correlation, which was positive or negative. Based on the identified clusters, the initial ARC model was revised. We then discuss the resulting model, outline future research, and derive implications for the larger context of the field of adaptive tutoring systems. Furthermore, potential benefits of the proposed concept are discussed and ethical issues are identified and addressed.

Palatability Assessment of Carbocysteine Oral Solution Strawberry Taste Versus Carbocysteine Oral Solution Mint Taste: A Blinded Randomized Study

Objective: To compare and evaluate the palatability of two carbocysteine oral solutions (strawberry vs. mint taste) among healthy children aged 2–12 years.

Methods: A randomized, triple-blind, crossover, palatability trial in 42 children aged 2–12 years. All subjects received two preparations of carbocysteine oral solutions (strawberry vs. mint) according to randomized administration sequences, and the administration process was recorded by video. The palatability assessed by emotional valences was performed using a facial action coding system by FaceReader™, which reflected the quantification degree of emotion; a positive value represents positive emotion, and a negative value represents negative emotion. At the same time, a face-to-face interview was conducted for 5- to 12-year-old participants. Then, the taste preferential rates were compared to assess the palatability of two carbocysteine oral solutions.

Results: Forty-two children were enrolled in this study. Twenty children first tasted the carbocysteine oral solution mint taste and then the strawberry taste preparation (M-S sequence), while 22 children tasted the strawberry preparation first and then the mint one (S-M sequence). The emotional valence of mint preparation (−0.9 in M-S and −1.2 in S-M) was both relatively lower than that of strawberry taste (both −0.7 in M-S and S-M) in two sequences; 69.0% (29/42) of participants’ emotional valences for strawberry preparation were higher than those for mint preparation. Among 27 participants aged ≥5 years, the taste preference rate was 88.5% (23/26) for the strawberry preparation (one missing value for the taste preference), and 77.8% of participants (21/27) chose the strawberry preparation if they had to take the medicine one more time.

Conclusion: The carbocysteine oral solution with strawberry taste is an appealing preparation since it was better received by children. The facial action coding system could be an effective alternative for palatability assessment of pediatric pharmaceutical products.

What's in a face: Automatic facial coding of untrained study participants compared to standardized inventories

Automatic facial coding (AFC) is a novel research tool to automatically analyze emotional facial expressions. AFC can classify emotional expressions with high accuracy in standardized picture inventories of intensively posed and prototypical expressions. However, classification of facial expressions of untrained study participants is more error prone. This discrepancy requires a direct comparison between these two sources of facial expressions. To this end, 70 untrained participants were asked to express joy, anger, surprise, sadness, disgust, and fear in a typical laboratory setting. Recorded videos were scored with a well-established AFC software (FaceReader, Noldus Information Technology). These were compared with AFC measures of standardized pictures from 70 trained actors (i.e., standardized inventories). We report the probability estimates of specific emotion categories and, in addition, Action Unit (AU) profiles for each emotion. Based on this, we used a novel machine learning approach to determine the relevant AUs for each emotion, separately for both datasets. First, misclassification was more frequent for some emotions of untrained participants. Second, AU intensities were generally lower in pictures of untrained participants compared to standardized pictures for all emotions. Third, although profiles of relevant AU overlapped substantially across the two data sets, there were also substantial differences in their AU profiles. This research provides evidence that the application of AFC is not limited to standardized facial expression inventories but can also be used to code facial expressions of untrained participants in a typical laboratory setting.

Assessment of the Psychophysiological State of Female Operators Under Simulated Microgravity

The article describes methods of non-verbal speech characteristics analysis used to determine psychophysiological state of female subjects under simulated microgravity conditions ("dry" immersion, DI), as well as the results of the study. A number of indicators of the acute period of adaptation to microgravity conditions was described. The acute adaptation period in female subjects began earlier (evening of the 1st day of DI) and ended faster than in male ones in previous studies (2nd day of DI). This was indicated by a decrease in the level of state anxiety (STAI, p < 0,05) and depression-dejection [Profile of Mood States (POMS), p < 0,05], as well as a decrease in pitch (p < 0,05) and voice intensity (p < 0,05). In addition, women, apparently, used the "freeze" coping strategy - the proportion of neutral facial expressions on the most intense days of the experiment was at maximum. The subjects in this experiment assessed their feelings and emotions better, giving more accurate answers in self-assessment questionnaires, but at the same time tried to look and sound as calm and confident as possible, controlling their expressions. Same trends in the subjects' cognitive performance were identified as in similar experimental conditions earlier: the subjects' psychophysiological excitement corresponded to better performance in sensorimotor tasks. The difference was in the speed of mathematical computation: women in the present study performed the computation faster on the same days when they made fewer pauses in speech, while in men in previous experiments this relationship was inverse.

Eye-Tracker Study of Influence of Affective Disruptive Content on User's Visual Attention and Emotional State

When reading interesting content or searching for information on a website, the appearance of a pop-up advertisement in the middle of the screen is perceived as irritating by a recipient. Interrupted cognitive processes are considered unwanted by the user but desired by advertising providers. Diverting visual attention away from the main content is intended to focus the user on the appeared disruptive content. Is the attempt to reach the user by any means justified? In this study, we examined the impact of pop-up emotional content on user reactions. For this purpose, a cognitive experiment was designed where a text-reading task was interrupted by two types of affective pictures: positive and negative ones. To measure the changes in user reactions, an eye-tracker (for analysis of eye movements and changes in gaze points) and an iMotion Platform (for analysis of face muscles' movements) were used. The results confirm the impact of the type of emotional content on users' reactions during cognitive process interruptions and indicate that the negative impact of cognitive process interruptions on the user can be reduced. The negative content evoked lower cognitive load, narrower visual attention, and lower irritation compared to positive content. These results offer insight on how to provide more efficient Internet advertising.

Creation and validation of the Picture-Set of Young Children's Affective Facial Expressions (PSYCAFE)

The immediate detection and correct processing of affective facial expressions are one of the most important competences in social interaction and thus a main subject in emotion and affect research. Generally, studies in these research domains, use pictures of adults who display affective facial expressions as experimental stimuli. However, for studies investigating developmental psychology and attachment behaviour it is necessary to use age-matched stimuli, where it is children that display affective expressions. PSYCAFE represents a newly developed picture-set of children’s faces. It includes reference portraits of girls and boys aged 4 to 6 years averaged digitally from different individual pictures, that were categorized to six basic affects (fear, disgust, happiness, sadness, anger and surprise) plus a neutral facial expression by cluster analysis. This procedure led to deindividualized and affect prototypical portraits. Individual affect expressive portraits of adults from an already validated picture-set (KDEF) were used in a similar way to create affect prototypical images also of adults. The stimulus set has been validated on human observers and entail emotion recognition accuracy rates and scores for intensity, authenticity and likeability ratings of the specific affect displayed. Moreover, the stimuli have also been characterized by the iMotions Facial Expression Analysis Module, providing additional data on probability values representing the likelihood that the stimuli depict the expected affect. Finally, the validation data from human observers and iMotions are compared to data on facial mimicry of healthy adults in response to these portraits, measured by facial EMG (m. zygomaticus major and m. corrugator supercilii).

External Communication of Autonomous Crews Under Simulation of Interplanetary Missions

Two experiments, with 17-day and 120-day isolation, were carried out within the frame of the Scientific International Research in Unique Terrestrial Station (SIRIUS) international project at the Institute of Biomedical Problems (Moscow, Russia). Manifestations of the "detachment" phenomenon in the crew - mission control center (MCC) communication previously identified in the Mars-500 project were confirmed in this study. As in the Mars-500 experiment, in the SIRIUS-19, the landing simulation in the halfway of isolation caused a temporary increase of crew communication with MCC. We also revealed several differences in the communication styles of male and female crew members. By the end of the experiment, there was a convergence of communication styles of all the SIRIUS crew members and also an increase in crew cohesion.

Applying Implicit Association Test Techniques and Facial Expression Analyses in the Comparative Evaluation of Website User Experience

This research project has the goal to verify whether the application of neuromarketing techniques, such as implicit association test (IAT) techniques and emotional facial expressions analyses may contribute to the assessment of user experience (UX) during and after website navigation. These techniques have been widely and positively applied in assessing customer experience (CX); however, little is known about their simultaneous application in the field of UX. As a specific context, the experience raised by different websites from two well-known automotive brands was compared. About 160 Italian university students were enrolled in an online experimental study. Participants performed a Brand Association Reaction Time Test (BARTT) version of the IAT where the two brands were compared according to different semantic dimensions already used in the automotive field. After completing the BARTT test, the participants navigated the target website: 80 participants navigated the first brand website, while the other half navigated the second brand website (between-subject design). During the first 3 min of website navigation, emotional facial expressions were recorded. The participants were asked to freely navigate the website home page, look for a car model and its characteristics and price, use the customising tool, and in the end, look for assistance. After the website navigation, all the participants performed, a second time, the BARTT version of the IAT, where the two brands were compared again, this time to assess whether the website navigation may impact the Implicit Associations previously detected. A traditional evaluation of the two websites was carried on by means of the classic heuristic evaluation. Findings from this study show, first of all, the significant results provided by neuromarketing techniques in the field of UX, as IAT can provide a positive application for assessing UX played by brand websites, thanks to the comparison of eventual changes in time reaction between the test performed before and after website navigation exposure. Secondly, results from emotional facial expression analyses during the navigation of both brand websites showed significant differences between the two brands, allowing the researchers to predict the emotional impact raised by each website. Finally, the positive correlation with heuristic evaluation shows that neuromarketing can be successfully applied in UX.

Prediction of Communication Effectiveness During Media Skills Training Using Commercial Automatic Non-verbal Recognition Systems

It is well recognised that social signals play an important role in communication effectiveness. Observation of videos to understand non-verbal behaviour is time-consuming and limits the potential to incorporate detailed and accurate feedback of this behaviour in practical applications such as communication skills training or performance evaluation. The aim of the current research is twofold: (1) to investigate whether off-the-shelf emotion recognition technology can detect social signals in media interviews and (2) to identify which combinations of social signals are most promising for evaluating trainees' performance in a media interview. To investigate this, non-verbal signals were automatically recognised from practice on-camera media interviews conducted within a media training setting with a sample size of 34. Automated non-verbal signal detection consists of multimodal features including facial expression, hand gestures, vocal behaviour and 'honest' signals. The on-camera interviews were categorised into effective and poor communication exemplars based on communication skills ratings provided by trainers and neutral observers which served as a ground truth. A correlation-based feature selection method was used to select signals associated with performance. To assess the accuracy of the selected features, a number of machine learning classification techniques were used. Naive Bayes analysis produced the best results with an F-measure of 0.76 and prediction accuracy of 78%. Results revealed that a combination of body movements, hand movements and facial expression are relevant for establishing communication effectiveness in the context of media interviews. The results of the current study have implications for the automatic evaluation of media interviews with a number of potential application areas including enhancing communication training including current media skills training.

Building and validation of a set of facial expression images to detect emotions: a transcultural study

The automatic emotion recognition from facial expressions has become an exceptional tool in research involving human subjects and has made it possible to obtain objective measurements of the emotional state of research subjects. Different software and commercial solutions are offered to perform this task. However, the adaptation to cultural context and the recognition of complex expressions and/or emotions are two of the main challenges faced by these solutions. Here, we describe the construction and validation of a set of facial expression images suitable for training a recognition system. Our datasets consist of images of people with no experience in acting who were recorded with a webcam as they performed a computer-assisted task in a room with a light background and overhead illumination. The six basic emotions and mockery were included and a combination of OpenCV, Dlib and Scikit-learn Python libraries were used to develop a support vector machine classifier. The code is available at GitHub and the images will be provided upon request. Since transcultural facial expressions to evaluate complex emotions and open-source solutions were used in this study, we strongly believe that our dataset will be useful in different research contexts.

Improving Human-Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression

An intriguing challenge in the human–robot interaction field is the prospect of endowing robots with emotional intelligence to make the interaction more genuine, intuitive, and natural. A crucial aspect in achieving this goal is the robot’s capability to infer and interpret human emotions. Thanks to its design and open programming platform, the NAO humanoid robot is one of the most widely used agents for human interaction. As with person-to-person communication, facial expressions are the privileged channel for recognizing the interlocutor’s emotional expressions. Although NAO is equipped with a facial expression recognition module, specific use cases may require additional features and affective computing capabilities that are not currently available. This study proposes a highly accurate convolutional-neural-network-based facial expression recognition model that is able to further enhance the NAO robot’ awareness of human facial expressions and provide the robot with an interlocutor’s arousal level detection capability. Indeed, the model tested during human–robot interactions was 91% and 90% accurate in recognizing happy and sad facial expressions, respectively; 75% accurate in recognizing surprised and scared expressions; and less accurate in recognizing neutral and angry expressions. Finally, the model was successfully integrated into the NAO SDK, thus allowing for high-performing facial expression classification with an inference time of 0.34 ± 0.04 s.

A Challenge-Based Learning Experience in Industrial Engineering in the Framework of Education 4.0

Current tendencies of product, project and services development focus on a higher consideration of the User Experience (UX). Therefore, traditional training and teaching methodologies need to adapt to prepare the students to develop strategies for problem solving for their professional education. Such needs have risen and interest in tendencies such as education and Industry 4.0 has grown. This paper presents and analyzes the process and results of a teaching implementation methodology based on Challenge-Based Learning (CBL). The paper describes the process followed, explaining the methodology precedents that led to the final implementation case. It also mentions previous experiments on product analysis and home automation developments that are linked to implementation of the technology. This case’s implementation, analysis and experimentation integrated the use of Emotional Domotics (ED) Tools for the UX analysis, to grant feedback and compare the students’ results with the bio-metrical and emotional computational analysis. The methodology, described through this document, allowed the students to better understand and experience some of the implications of an interconnected system with instant information feedback. This allowed them to better grasp part of the impact that the tendency towards the Internet of Things (IoT) is currently having, and the impact of the improvement proposals from the students.