1
|
Bradley ER, Portanova J, Woolley JD, Buck B, Painter IS, Hankin M, Xu W, Cohen T. Quantifying abnormal emotion processing: A novel computational assessment method and application in schizophrenia. Psychiatry Res 2024; 336:115893. [PMID: 38657475 DOI: 10.1016/j.psychres.2024.115893] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 12/31/2023] [Accepted: 04/03/2024] [Indexed: 04/26/2024]
Abstract
Abnormal emotion processing is a core feature of schizophrenia spectrum disorders (SSDs) that encompasses multiple operations. While deficits in some areas have been well-characterized, we understand less about abnormalities in the emotion processing that happens through language, which is highly relevant for social life. Here, we introduce a novel method using deep learning to estimate emotion processing rapidly from spoken language, testing this approach in male-identified patients with SSDs (n = 37) and healthy controls (n = 51). Using free responses to evocative stimuli, we derived a measure of appropriateness, or "emotional alignment" (EA). We examined psychometric characteristics of EA and its sensitivity to a single-dose challenge of oxytocin, a neuropeptide shown to enhance the salience of socioemotional information in SSDs. Patients showed impaired EA relative to controls, and impairment correlated with poorer social cognitive skill and more severe motivation and pleasure deficits. Adding EA to a logistic regression model with language-based measures of formal thought disorder (FTD) improved classification of patients versus controls. Lastly, oxytocin administration improved EA but not FTD among patients. While additional validation work is needed, these initial results suggest that an automated assay using spoken language may be a promising approach to assess emotion processing in SSDs.
Collapse
Affiliation(s)
- Ellen R Bradley
- Department of Psychiatry and Behavioral Sciences, Weill Institute for Neurosciences, University of California, San Francisco, CA, USA; San Francisco Veterans Affairs Medical Center, CA, USA.
| | - Jake Portanova
- Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA
| | - Josh D Woolley
- Department of Psychiatry and Behavioral Sciences, Weill Institute for Neurosciences, University of California, San Francisco, CA, USA; San Francisco Veterans Affairs Medical Center, CA, USA
| | - Benjamin Buck
- Behavioral Research in Technology and Engineering (BRiTE) Center, Department of Psychiatry and Behavioral Sciences, University of Washington, USA
| | - Ian S Painter
- Department of Statistics, University of Washington, USA
| | | | - Weizhe Xu
- Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA
| | - Trevor Cohen
- Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA; Behavioral Research in Technology and Engineering (BRiTE) Center, Department of Psychiatry and Behavioral Sciences, University of Washington, USA
| |
Collapse
|
2
|
Hall NT, Hallquist MN, Martin EA, Lian W, Jonas KG, Kotov R. Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders. Proc Natl Acad Sci U S A 2024; 121:e2313665121. [PMID: 38530896 PMCID: PMC10998559 DOI: 10.1073/pnas.2313665121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Accepted: 01/18/2024] [Indexed: 03/28/2024] Open
Abstract
Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.
Collapse
Affiliation(s)
- Nathan T. Hall
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC27599
| | - Michael N. Hallquist
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC27599
| | - Elizabeth A. Martin
- Department of Psychological Science, University of California, Irvine, CA92697
| | - Wenxuan Lian
- Department of Psychiatry, Stony Brook University, Stoney Brook, NY11794
| | | | - Roman Kotov
- Department of Psychiatry, Stony Brook University, Stoney Brook, NY11794
| |
Collapse
|
3
|
Gupta T, Horton WS, Haase CM, Carol EE, Mittal VA. Clues from caregiver emotional language usage highlight the link between putative social environment and the psychosis-risk syndrome. Schizophr Res 2023; 259:4-10. [PMID: 35400558 PMCID: PMC9578001 DOI: 10.1016/j.schres.2022.03.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Revised: 03/23/2022] [Accepted: 03/24/2022] [Indexed: 10/18/2022]
Abstract
Familial emotional word usage has long been implicated in symptom progression in schizophrenia. However, few studies have examined caregiver emotional word usage prior to the onset of psychosis, among those with a clinical high-risk (CHR) syndrome. The current study examined emotional word usage in a sample of caregivers of CHR individuals (N = 37) and caregivers of healthy controls (N = 40) and links with clinical symptoms in CHR individuals. Caregivers completed a speech sample task in which they were asked to speak about the participant; speech samples were then transcribed and analyzed for general positive (e.g. good) and negative (e.g., worthless) emotional words as well as words expressing three specific negative emotions (i.e., anxiety, anger, and sadness) using Linguistic Inquiry and Word Count (LIWC). Findings indicated that (1) CHR caregivers used more negative and anxiety words compared to control caregivers; and (2) less positive word usage among CHR caregivers were related to more positive symptomatology among CHR individuals. These findings point toward the utility of automated language analysis in assessing the intersections between caregiver emotional language use and psychopathology.
Collapse
Affiliation(s)
- Tina Gupta
- Department of Psychology, Northwestern University, Evanston, IL, USA.
| | - William S Horton
- Department of Psychology, Northwestern University, Evanston, IL, USA
| | - Claudia M Haase
- Department of Psychology, Northwestern University, Evanston, IL, USA; School of Education and Social Policy, Northwestern University, Evanston, IL, USA
| | - Emily E Carol
- Psychotic Disorders Division, McLean Hospital/Harvard Medical School, Belmont, MA, USA
| | - Vijay A Mittal
- Department of Psychology, Northwestern University, Evanston, IL, USA
| |
Collapse
|
4
|
Shore D, Robertson O, Lafit G, Parkinson B. Facial Regulation During Dyadic Interaction: Interpersonal Effects on Cooperation. AFFECTIVE SCIENCE 2023; 4:506-516. [PMID: 37744968 PMCID: PMC10514003 DOI: 10.1007/s42761-023-00208-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 07/18/2023] [Indexed: 09/26/2023]
Abstract
This study investigated interpersonal effects of regulating naturalistic facial signals on cooperation during an iterative Prisoner's Dilemma (IPD) game. Fifty pairs of participants played ten IPD rounds across a video link then reported on their own and their partner's expressed emotion and facial regulation in a video-cued recall (VCR) procedure. iMotions software allowed us to auto-code actors' and partners' facial activity following the outcome of each round. We used two-level mixed effects logistic regression to assess over-time actor and partner effects of auto-coded facial activity, self-reported facial regulation, and perceptions of the partner's facial regulation on the actor's subsequent cooperation. Actors were significantly less likely to cooperate when their partners had defected on the previous round. None of the lagged scores based on auto-coded facial activity were significant predictors of cooperation. However, VCR variables representing partner's positive regulation of expressions and actor's perception of partner's positive regulation both significantly increased the probability of subsequent actor cooperation after controlling for prior defection. These results offer preliminary evidence about interpersonal effects of facial regulation in interactive contexts and illustrate how dynamic dyadic emotional processes can be systematically investigated in controlled settings. Supplementary Information The online version contains supplementary material available at 10.1007/s42761-023-00208-y.
Collapse
Affiliation(s)
- Danielle Shore
- Department of Experimental Psychology, University of Oxford, New Radcliffe House, Oxford, OX6 2RR UK
| | - Olly Robertson
- Department of Experimental Psychology, University of Oxford, New Radcliffe House, Oxford, OX6 2RR UK
- Department of Psychiatry, University of Oxford, Oxford, UK
| | - Ginette Lafit
- Research Group for Quantitative Psychology and Individual Differences, KU Leuven, Leuven, Belgium
- Center for Contextual Psychiatry, Department of Neuroscience, KU Leuven, Leuven, Belgium
| | - Brian Parkinson
- Department of Experimental Psychology, University of Oxford, New Radcliffe House, Oxford, OX6 2RR UK
| |
Collapse
|
5
|
Gupta T, Osborne KJ, Nadig A, Haase CM, Mittal VA. Alterations in facial expressions in individuals at risk for psychosis: a facial electromyography approach using emotionally evocative film clips. Psychol Med 2023; 53:5829-5838. [PMID: 36285533 PMCID: PMC10130238 DOI: 10.1017/s0033291722003087] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
BACKGROUND Negative symptoms such as blunted facial expressivity are characteristic of schizophrenia. However, it is not well-understood if and what abnormalities are present in individuals at clinical high-risk (CHR) for psychosis. METHODS This experimental study employed facial electromyography (left zygomaticus major and left corrugator supercilia) in a sample of CHR individuals (N = 34) and healthy controls (N = 32) to detect alterations in facial expressions in response to emotionally evocative film clips and to determine links with symptoms. RESULTS Findings revealed that the CHR group showed facial blunting manifested in reduced zygomatic activity in response to an excitement (but not amusement, fear, or sadness) film clip compared to controls. Reductions in zygomatic activity in the CHR group emerged in response to the emotionally evocative peak period of the excitement film clip. Lower zygomaticus activity during the excitement clip was related to anxiety while lower rates of change in zygomatic activity during the excitement video clip were related to higher psychosis risk conversion scores. CONCLUSIONS Together, these findings inform vulnerability/disease driving mechanisms and biomarker and treatment development.
Collapse
Affiliation(s)
- Tina Gupta
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, USA
| | - K. Juston Osborne
- Department of Psychology, Northwestern University, Evanston, IL, USA
| | - Ajay Nadig
- Harvard/MIT MD-PhD Program, Harvard Medical School, Boston, MA, 02115
| | - Claudia M. Haase
- Department of Psychology, Northwestern University, Evanston, IL, USA
- School of Education and Social Policy, Northwestern University, Evanston, IL, USA
| | - Vijay A. Mittal
- Department of Psychology, Northwestern University, Evanston, IL, USA
| |
Collapse
|
6
|
Loch AA, Gondim JM, Argolo FC, Lopes-Rocha AC, Andrade JC, van de Bilt MT, de Jesus LP, Haddad NM, Cecchi GA, Mota NB, Gattaz WF, Corcoran CM, Ara A. Detecting at-risk mental states for psychosis (ARMS) using machine learning ensembles and facial features. Schizophr Res 2023; 258:45-52. [PMID: 37473667 PMCID: PMC10448183 DOI: 10.1016/j.schres.2023.07.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Revised: 04/26/2023] [Accepted: 07/10/2023] [Indexed: 07/22/2023]
Abstract
AIMS Our study aimed to develop a machine learning ensemble to distinguish "at-risk mental states for psychosis" (ARMS) subjects from control individuals from the general population based on facial data extracted from video-recordings. METHODS 58 non-help-seeking medication-naïve ARMS and 70 healthy subjects were screened from a general population sample. At-risk status was assessed with the Structured Interview for Prodromal Syndromes (SIPS), and "Subject's Overview" section was filmed (5-10 min). Several features were extracted, e.g., eye and mouth aspect ratio, Euler angles, coordinates from 51 facial landmarks. This elicited 649 facial features, which were further selected using Gradient Boosting Machines (AdaBoost combined with Random Forests). Data was split in 70/30 for training, and Monte Carlo cross validation was used. RESULTS Final model reached 83 % of mean F1-score, and balanced accuracy of 85 %. Mean area under the curve for the receiver operator curve classifier was 93 %. Convergent validity testing showed that two features included in the model were significantly correlated with Avolition (SIPS N2 item) and expression of emotion (SIPS N3 item). CONCLUSION Our model capitalized on short video-recordings from individuals recruited from the general population, effectively distinguishing between ARMS and controls. Results are encouraging for large-screening purposes in low-resource settings.
Collapse
Affiliation(s)
- Alexandre Andrade Loch
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil; Instituto Nacional de Biomarcadores em Neuropsiquiatria (INBION), Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil.
| | - João Medrado Gondim
- Instituto de Computação, Universidade Federal da Bahia, Salvador, BA, Brazil
| | - Felipe Coelho Argolo
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil
| | - Ana Caroline Lopes-Rocha
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil
| | - Julio Cesar Andrade
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil
| | - Martinus Theodorus van de Bilt
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil; Instituto Nacional de Biomarcadores em Neuropsiquiatria (INBION), Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil
| | - Leonardo Peroni de Jesus
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil
| | - Natalia Mansur Haddad
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil
| | | | - Natalia Bezerra Mota
- Instituto de Psiquiatria (IPUB), Departamento de Psiquiatria e Medicina Legal, Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro, Brazil; Research Department at Motrix Lab - Motrix, Rio de Janeiro, Brazil
| | - Wagner Farid Gattaz
- Laboratório de Neurociencias (LIM 27), Instituto de Psiquiatria, Hospital das Clínicas HCFMUSP, Faculdade de Medicina, Universidade de Sao Paulo, Sao Paulo, SP, Brazil; Instituto Nacional de Biomarcadores em Neuropsiquiatria (INBION), Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil
| | - Cheryl Mary Corcoran
- Icahn School of Medicine at Mount Sinai, New York, NY, USA; James J. Peters VA Medical Center Bronx, NY, USA
| | - Anderson Ara
- Statistics Department, Federal University of Paraná, Curitiba, PR, Brazil
| |
Collapse
|
7
|
Ricard JR, Gupta T, Vargas T, Haase CM, Mittal VA. Genuine and non-genuine smiles in individuals meeting criteria for a clinical high-risk syndrome. Early Interv Psychiatry 2022; 16:875-882. [PMID: 34725928 PMCID: PMC9056581 DOI: 10.1111/eip.13233] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/28/2021] [Revised: 08/21/2021] [Accepted: 10/19/2021] [Indexed: 01/17/2023]
Abstract
AIM Psychosis is characterized by both alterations in emotional functioning and environmental stressors including bullying victimization. Recent evidence suggests that some alterations in emotional functioning (e.g., blunted positive facial expressions) are already present in the psychosis risk period. Yet, some clinically relevant facial emotions have not been investigated such as genuine smiles (thought to reflect genuine positive emotions) and non-genuine smiles (thought to fake positive or mask negative emotions) in individuals meeting criteria for a clinical high-risk (CHR) syndrome. Further, despite a compelling conceptual basis to suggest a link between affective expression and exposure to environmental stress, to date, no investigations have sought to examine this association. Here, we aim to assess differences between a sample of CHR (N = 65) and control (N = 67) individuals in genuine and non-genuine smiles and associations with bullying victimization. METHODS Smiles (i.e., genuine; non-genuine) were objectively coded on a second-by-second basis using the Facial Action Coding System during a digitally recorded clinical interview segment. Bullying victimization was measured via parent report. RESULTS Findings revealed that the CHR group (1) showed blunted genuine (but not non-genuine) smiles compared to controls. Moreover, (2) bullying victimization was related to blunted genuine smiles, but not non-genuine smiles. CONCLUSION These findings expand our understanding of emotional alterations in this group with implications for diagnosis (highlighting blunted genuine smiles as a specific marker) and etiology (underscoring the role of bullying victimization in the etiology of emotional dysfunction).
Collapse
Affiliation(s)
- Jordyn R Ricard
- School of Education and Social Policy, Northwestern University, Evanston, Illinois, USA
| | - Tina Gupta
- Department of Psychology, Northwestern University, Evanston, Illinois, USA
| | - Teresa Vargas
- Department of Psychology, Northwestern University, Evanston, Illinois, USA
| | - Claudia M Haase
- School of Education and Social Policy and (by courtesy) Department of Psychology, Northwestern University, Evanston, Illinois, USA
| | - Vijay A Mittal
- Department of Psychology, Department of Psychiatry, Institute for Policy Research (IPR), Department of Medical Social Sciences, and Institute for Innovations in Developmental Sciences (DevSci), Northwestern University, Evanston, Illinois, USA
| |
Collapse
|
8
|
Cross MP, Acevedo AM, Leger KA, Pressman SD. How and Why Could Smiling Influence Physical Health? A Conceptual Review. Health Psychol Rev 2022; 17:321-343. [PMID: 35285408 DOI: 10.1080/17437199.2022.2052740] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Smiling has been a topic of interest to psychologists for decades, with a myriad of studies tying this behavior to well-being. Despite this, we know surprisingly little about the nature of the connections between smiling and physical health. We review the literature connecting both naturally occurring smiles and experimentally manipulated smiles to physical health and health-relevant outcomes. This work is discussed in the context of existing affect and health-relevant theoretical models that help explain the connection between smiling and physical health including the facial feedback hypothesis, the undoing hypothesis, the generalized unsafety theory of stress, and polyvagal theory. We also describe a number of plausible pathways, some new and relatively untested, through which smiling may influence physical health such as trait or state positive affect, social relationships, stress buffering, and the oculocardiac reflex. Finally, we provide a discussion of possible future directions, including the importance of cultural variation and replication. Although this field is still in its infancy, the findings from both naturally occurring smile studies and experimentally manipulated smile studies consistently suggest that smiling may have a number of health-relevant benefits including beneficially impacting our physiology during acute stress, improved stress recovery, and reduced illness over time.
Collapse
Affiliation(s)
- Marie P Cross
- Department of Biobehavioral Health, Pennsylvania State University, University Park, University Park, PA, USA
| | - Amanda M Acevedo
- Department of Psychological Science, University of California, Irvine, Irvine, CA, USA
| | - Kate A Leger
- Department of Psychology, University of Kentucky, Lexington, Lexington, KY, USA
| | - Sarah D Pressman
- Department of Psychological Science, University of California, Irvine, Irvine, CA, USA
| |
Collapse
|
9
|
Computerized analysis of facial expressions in serious mental illness. Schizophr Res 2022; 241:44-51. [PMID: 35074531 PMCID: PMC8978090 DOI: 10.1016/j.schres.2021.12.026] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/06/2021] [Revised: 11/19/2021] [Accepted: 12/18/2021] [Indexed: 12/30/2022]
Abstract
Blunted facial affect is a transdiagnostic component of Serious Mental Illness (SMI) and is associated with a host of negative outcomes. However, blunted facial affect is a poorly understood phenomenon, with no known cures or treatments. A critical step in better understanding its phenotypic expression involves clarifying which facial expressions are altered in specific ways and under what contexts. The current literature suggests that individuals with SMI show decreased positive facial expressions, but typical, or even increased negative facial expressions during laboratory tasks. While this literature has coalesced around general trends, significantly more nuance is available regarding what components facial expressions are atypical and how those components are associated with increased severity of clinical ratings. The present project leveraged computerized facial analysis to test whether clinician-rated blunted affect is driven by decreases in duration, intensity, or frequency of positive versus other facial expressions during a structured clinical interview. Stable outpatients meeting criteria for SMI (N = 59) were examined. Facial expression did not generally vary as a function of clinical diagnosis. Overall, clinically-rated blunted affect was not associated with positive expressions, but was associated with decreased surprise and increased anger, sadness, and fear expressions. Blunted affect is not a monolithic lack of expressivity, and increased precision in operationally defining it is critical for uncovering its causes and maintaining factors. Our discussion focuses on this effort, and on advancing digital phenotyping of blunted facial affect more generally.
Collapse
|