Alterations in facial expressions of emotion: Determining the promise of ultrathin slicing approaches and comparing human and automated coding methods in psychosis risk

Quantifying abnormal emotion processing: A novel computational assessment method and application in schizophrenia

Abnormal emotion processing is a core feature of schizophrenia spectrum disorders (SSDs) that encompasses multiple operations. While deficits in some areas have been well-characterized, we understand less about abnormalities in the emotion processing that happens through language, which is highly relevant for social life. Here, we introduce a novel method using deep learning to estimate emotion processing rapidly from spoken language, testing this approach in male-identified patients with SSDs (n = 37) and healthy controls (n = 51). Using free responses to evocative stimuli, we derived a measure of appropriateness, or "emotional alignment" (EA). We examined psychometric characteristics of EA and its sensitivity to a single-dose challenge of oxytocin, a neuropeptide shown to enhance the salience of socioemotional information in SSDs. Patients showed impaired EA relative to controls, and impairment correlated with poorer social cognitive skill and more severe motivation and pleasure deficits. Adding EA to a logistic regression model with language-based measures of formal thought disorder (FTD) improved classification of patients versus controls. Lastly, oxytocin administration improved EA but not FTD among patients. While additional validation work is needed, these initial results suggest that an automated assay using spoken language may be a promising approach to assess emotion processing in SSDs.

Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders

Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.

Clues from caregiver emotional language usage highlight the link between putative social environment and the psychosis-risk syndrome

Familial emotional word usage has long been implicated in symptom progression in schizophrenia. However, few studies have examined caregiver emotional word usage prior to the onset of psychosis, among those with a clinical high-risk (CHR) syndrome. The current study examined emotional word usage in a sample of caregivers of CHR individuals (N = 37) and caregivers of healthy controls (N = 40) and links with clinical symptoms in CHR individuals. Caregivers completed a speech sample task in which they were asked to speak about the participant; speech samples were then transcribed and analyzed for general positive (e.g. good) and negative (e.g., worthless) emotional words as well as words expressing three specific negative emotions (i.e., anxiety, anger, and sadness) using Linguistic Inquiry and Word Count (LIWC). Findings indicated that (1) CHR caregivers used more negative and anxiety words compared to control caregivers; and (2) less positive word usage among CHR caregivers were related to more positive symptomatology among CHR individuals. These findings point toward the utility of automated language analysis in assessing the intersections between caregiver emotional language use and psychopathology.

Facial Regulation During Dyadic Interaction: Interpersonal Effects on Cooperation

This study investigated interpersonal effects of regulating naturalistic facial signals on cooperation during an iterative Prisoner's Dilemma (IPD) game. Fifty pairs of participants played ten IPD rounds across a video link then reported on their own and their partner's expressed emotion and facial regulation in a video-cued recall (VCR) procedure. iMotions software allowed us to auto-code actors' and partners' facial activity following the outcome of each round. We used two-level mixed effects logistic regression to assess over-time actor and partner effects of auto-coded facial activity, self-reported facial regulation, and perceptions of the partner's facial regulation on the actor's subsequent cooperation. Actors were significantly less likely to cooperate when their partners had defected on the previous round. None of the lagged scores based on auto-coded facial activity were significant predictors of cooperation. However, VCR variables representing partner's positive regulation of expressions and actor's perception of partner's positive regulation both significantly increased the probability of subsequent actor cooperation after controlling for prior defection. These results offer preliminary evidence about interpersonal effects of facial regulation in interactive contexts and illustrate how dynamic dyadic emotional processes can be systematically investigated in controlled settings.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-023-00208-y.

Alterations in facial expressions in individuals at risk for psychosis: a facial electromyography approach using emotionally evocative film clips

BACKGROUND

Negative symptoms such as blunted facial expressivity are characteristic of schizophrenia. However, it is not well-understood if and what abnormalities are present in individuals at clinical high-risk (CHR) for psychosis.

METHODS

This experimental study employed facial electromyography (left zygomaticus major and left corrugator supercilia) in a sample of CHR individuals (N = 34) and healthy controls (N = 32) to detect alterations in facial expressions in response to emotionally evocative film clips and to determine links with symptoms.

RESULTS

Findings revealed that the CHR group showed facial blunting manifested in reduced zygomatic activity in response to an excitement (but not amusement, fear, or sadness) film clip compared to controls. Reductions in zygomatic activity in the CHR group emerged in response to the emotionally evocative peak period of the excitement film clip. Lower zygomaticus activity during the excitement clip was related to anxiety while lower rates of change in zygomatic activity during the excitement video clip were related to higher psychosis risk conversion scores.

CONCLUSIONS

Together, these findings inform vulnerability/disease driving mechanisms and biomarker and treatment development.

Detecting at-risk mental states for psychosis (ARMS) using machine learning ensembles and facial features

AIMS

Our study aimed to develop a machine learning ensemble to distinguish "at-risk mental states for psychosis" (ARMS) subjects from control individuals from the general population based on facial data extracted from video-recordings.

METHODS

58 non-help-seeking medication-naïve ARMS and 70 healthy subjects were screened from a general population sample. At-risk status was assessed with the Structured Interview for Prodromal Syndromes (SIPS), and "Subject's Overview" section was filmed (5-10 min). Several features were extracted, e.g., eye and mouth aspect ratio, Euler angles, coordinates from 51 facial landmarks. This elicited 649 facial features, which were further selected using Gradient Boosting Machines (AdaBoost combined with Random Forests). Data was split in 70/30 for training, and Monte Carlo cross validation was used.

RESULTS

Final model reached 83 % of mean F1-score, and balanced accuracy of 85 %. Mean area under the curve for the receiver operator curve classifier was 93 %. Convergent validity testing showed that two features included in the model were significantly correlated with Avolition (SIPS N2 item) and expression of emotion (SIPS N3 item).

CONCLUSION

Our model capitalized on short video-recordings from individuals recruited from the general population, effectively distinguishing between ARMS and controls. Results are encouraging for large-screening purposes in low-resource settings.

Genuine and non-genuine smiles in individuals meeting criteria for a clinical high-risk syndrome

AIM

Psychosis is characterized by both alterations in emotional functioning and environmental stressors including bullying victimization. Recent evidence suggests that some alterations in emotional functioning (e.g., blunted positive facial expressions) are already present in the psychosis risk period. Yet, some clinically relevant facial emotions have not been investigated such as genuine smiles (thought to reflect genuine positive emotions) and non-genuine smiles (thought to fake positive or mask negative emotions) in individuals meeting criteria for a clinical high-risk (CHR) syndrome. Further, despite a compelling conceptual basis to suggest a link between affective expression and exposure to environmental stress, to date, no investigations have sought to examine this association. Here, we aim to assess differences between a sample of CHR (N = 65) and control (N = 67) individuals in genuine and non-genuine smiles and associations with bullying victimization.

METHODS

Smiles (i.e., genuine; non-genuine) were objectively coded on a second-by-second basis using the Facial Action Coding System during a digitally recorded clinical interview segment. Bullying victimization was measured via parent report.

RESULTS

Findings revealed that the CHR group (1) showed blunted genuine (but not non-genuine) smiles compared to controls. Moreover, (2) bullying victimization was related to blunted genuine smiles, but not non-genuine smiles.

CONCLUSION

These findings expand our understanding of emotional alterations in this group with implications for diagnosis (highlighting blunted genuine smiles as a specific marker) and etiology (underscoring the role of bullying victimization in the etiology of emotional dysfunction).

How and Why Could Smiling Influence Physical Health? A Conceptual Review

Smiling has been a topic of interest to psychologists for decades, with a myriad of studies tying this behavior to well-being. Despite this, we know surprisingly little about the nature of the connections between smiling and physical health. We review the literature connecting both naturally occurring smiles and experimentally manipulated smiles to physical health and health-relevant outcomes. This work is discussed in the context of existing affect and health-relevant theoretical models that help explain the connection between smiling and physical health including the facial feedback hypothesis, the undoing hypothesis, the generalized unsafety theory of stress, and polyvagal theory. We also describe a number of plausible pathways, some new and relatively untested, through which smiling may influence physical health such as trait or state positive affect, social relationships, stress buffering, and the oculocardiac reflex. Finally, we provide a discussion of possible future directions, including the importance of cultural variation and replication. Although this field is still in its infancy, the findings from both naturally occurring smile studies and experimentally manipulated smile studies consistently suggest that smiling may have a number of health-relevant benefits including beneficially impacting our physiology during acute stress, improved stress recovery, and reduced illness over time.

Computerized analysis of facial expressions in serious mental illness

Blunted facial affect is a transdiagnostic component of Serious Mental Illness (SMI) and is associated with a host of negative outcomes. However, blunted facial affect is a poorly understood phenomenon, with no known cures or treatments. A critical step in better understanding its phenotypic expression involves clarifying which facial expressions are altered in specific ways and under what contexts. The current literature suggests that individuals with SMI show decreased positive facial expressions, but typical, or even increased negative facial expressions during laboratory tasks. While this literature has coalesced around general trends, significantly more nuance is available regarding what components facial expressions are atypical and how those components are associated with increased severity of clinical ratings. The present project leveraged computerized facial analysis to test whether clinician-rated blunted affect is driven by decreases in duration, intensity, or frequency of positive versus other facial expressions during a structured clinical interview. Stable outpatients meeting criteria for SMI (N = 59) were examined. Facial expression did not generally vary as a function of clinical diagnosis. Overall, clinically-rated blunted affect was not associated with positive expressions, but was associated with decreased surprise and increased anger, sadness, and fear expressions. Blunted affect is not a monolithic lack of expressivity, and increased precision in operationally defining it is critical for uncovering its causes and maintaining factors. Our discussion focuses on this effort, and on advancing digital phenotyping of blunted facial affect more generally.