1
|
Martin EA, Lian W, Oltmanns JR, Jonas KG, Samaras D, Hallquist MN, Ruggero CJ, Clouston SAP, Kotov R. Behavioral meaures of psychotic disorders: Using automatic facial coding to detect nonverbal expressions in video. J Psychiatr Res 2024; 176:9-17. [PMID: 38830297 DOI: 10.1016/j.jpsychires.2024.05.056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Revised: 04/11/2024] [Accepted: 05/29/2024] [Indexed: 06/05/2024]
Abstract
Emotional deficits in psychosis are prevalent and difficult to treat. In particular, much remains unknown about facial expression abnormalities, and a key reason is that expressions are very labor-intensive to code. Automatic facial coding (AFC) can remove this barrier. The current study sought to both provide evidence for the utility of AFC in psychosis for research purposes and to provide evidence that AFC are valid measures of clinical constructs. Changes of facial expressions and head position of participants-39 with schizophrenia/schizoaffective disorder (SZ), 46 with other psychotic disorders (OP), and 108 never psychotic individuals (NP)-were assessed via FaceReader, a commercially available automated facial expression analysis software, using video recorded during a clinical interview. We first examined the behavioral measures of the psychotic disorder groups and tested if they can discriminate between the groups. Next, we evaluated links of behavioral measures with clinical symptoms, controlling for group membership. We found the SZ group was characterized by significantly less variation in neutral expressions, happy expressions, arousal, and head movements compared to NP. These measures discriminated SZ from NP well (AUC = 0.79, sensitivity = 0.79, specificity = 0.67) but discriminated SZ from OP less well (AUC = 0.66, sensitivity = 0.77, specificity = 0.46). We also found significant correlations between clinician-rated symptoms and most behavioral measures (particularly happy expressions, arousal, and head movements). Taken together, these results suggest that AFC can provide useful behavioral measures of psychosis, which could improve research on non-verbal expressions in psychosis and, ultimately, enhance treatment.
Collapse
Affiliation(s)
- Elizabeth A Martin
- Department of Psychological Science, University of California, Irvine, CA, USA.
| | - Wenxuan Lian
- Department of Materials Science and Engineering and Department of Applied Math and Statistics, Stony Brook University, Stony Brook, NY, USA
| | - Joshua R Oltmanns
- Department of Psychiatry, Stony Brook University, Stony Brook, NY, USA
| | - Katherine G Jonas
- Department of Psychiatry, Stony Brook University, Stony Brook, NY, USA
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Michael N Hallquist
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Camilo J Ruggero
- Department of Psychology, University of Texas at Dallas, Richardson, TX, USA
| | - Sean A P Clouston
- Program in Public Health and Department of Family, Population, and Preventive Medicine, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA
| | - Roman Kotov
- Department of Psychiatry, Stony Brook University, Stony Brook, NY, USA.
| |
Collapse
|
2
|
Hall NT, Hallquist MN, Martin EA, Lian W, Jonas KG, Kotov R. Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders. Proc Natl Acad Sci U S A 2024; 121:e2313665121. [PMID: 38530896 PMCID: PMC10998559 DOI: 10.1073/pnas.2313665121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Accepted: 01/18/2024] [Indexed: 03/28/2024] Open
Abstract
Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.
Collapse
Affiliation(s)
- Nathan T. Hall
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC27599
| | - Michael N. Hallquist
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC27599
| | - Elizabeth A. Martin
- Department of Psychological Science, University of California, Irvine, CA92697
| | - Wenxuan Lian
- Department of Psychiatry, Stony Brook University, Stoney Brook, NY11794
| | | | - Roman Kotov
- Department of Psychiatry, Stony Brook University, Stoney Brook, NY11794
| |
Collapse
|
3
|
Lindroth H, Nalaie K, Raghu R, Ayala IN, Busch C, Bhattacharyya A, Moreno Franco P, Diedrich DA, Pickering BW, Herasevich V. Applied Artificial Intelligence in Healthcare: A Review of Computer Vision Technology Application in Hospital Settings. J Imaging 2024; 10:81. [PMID: 38667979 PMCID: PMC11050909 DOI: 10.3390/jimaging10040081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Revised: 03/08/2024] [Accepted: 03/11/2024] [Indexed: 04/28/2024] Open
Abstract
Computer vision (CV), a type of artificial intelligence (AI) that uses digital videos or a sequence of images to recognize content, has been used extensively across industries in recent years. However, in the healthcare industry, its applications are limited by factors like privacy, safety, and ethical concerns. Despite this, CV has the potential to improve patient monitoring, and system efficiencies, while reducing workload. In contrast to previous reviews, we focus on the end-user applications of CV. First, we briefly review and categorize CV applications in other industries (job enhancement, surveillance and monitoring, automation, and augmented reality). We then review the developments of CV in the hospital setting, outpatient, and community settings. The recent advances in monitoring delirium, pain and sedation, patient deterioration, mechanical ventilation, mobility, patient safety, surgical applications, quantification of workload in the hospital, and monitoring for patient events outside the hospital are highlighted. To identify opportunities for future applications, we also completed journey mapping at different system levels. Lastly, we discuss the privacy, safety, and ethical considerations associated with CV and outline processes in algorithm development and testing that limit CV expansion in healthcare. This comprehensive review highlights CV applications and ideas for its expanded use in healthcare.
Collapse
Affiliation(s)
- Heidi Lindroth
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- Center for Aging Research, Regenstrief Institute, School of Medicine, Indiana University, Indianapolis, IN 46202, USA
- Center for Health Innovation and Implementation Science, School of Medicine, Indiana University, Indianapolis, IN 46202, USA
| | - Keivan Nalaie
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Roshini Raghu
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
| | - Ivan N. Ayala
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
| | - Charles Busch
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- College of Engineering, University of Wisconsin-Madison, Madison, WI 53705, USA
| | | | - Pablo Moreno Franco
- Department of Transplantation Medicine, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Daniel A. Diedrich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Brian W. Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Vitaly Herasevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| |
Collapse
|
4
|
Cowan T, Phalen P, Brown CH, Blanchard J, Bennett M. We need to make progress on blunted affect: A commentary. Schizophr Res 2024; 264:263-265. [PMID: 38198877 DOI: 10.1016/j.schres.2024.01.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Revised: 12/14/2023] [Accepted: 01/01/2024] [Indexed: 01/12/2024]
Affiliation(s)
- T Cowan
- Department of Psychiatry, University of Maryland School of Medicine, 717 W. Lombard St. 5th Floor, Baltimore, MD 21201, United States of America
| | - P Phalen
- Department of Psychiatry, University of Maryland School of Medicine, 717 W. Lombard St. 5th Floor, Baltimore, MD 21201, United States of America
| | - C H Brown
- Department of Epidemiology, University of Maryland School of Medicine, 660 W. Redwood St., Baltimore, MD 21201, United States of America; VA Capital Healthcare Network Mental Illness Research, Education, and Clinical Center (MIRECC), Veterans Affairs Maryland Health Care System (Baltimore Annex), 209 West Fayette Street, Baltimore, MD 20210, United States of America
| | - J Blanchard
- Department of Psychology, University of Maryland College Park, Biology/Psychology Building, 4094 Campus Dr., College Park, MD 20742, United States of America
| | - M Bennett
- Department of Psychiatry, University of Maryland School of Medicine, 717 W. Lombard St. 5th Floor, Baltimore, MD 21201, United States of America; VA Capital Healthcare Network Mental Illness Research, Education, and Clinical Center (MIRECC), Veterans Affairs Maryland Health Care System (Baltimore Annex), 209 West Fayette Street, Baltimore, MD 20210, United States of America.
| |
Collapse
|
5
|
Cowan T, Rodriguez ZB, Strauss GP, Raugh IM, Cohen AS. Computerized analysis of facial expression reveals objective indices of blunted facial affect. Eur Arch Psychiatry Clin Neurosci 2023:10.1007/s00406-023-01696-6. [PMID: 37878034 DOI: 10.1007/s00406-023-01696-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/01/2023] [Accepted: 09/20/2023] [Indexed: 10/26/2023]
Abstract
Blunted affect is associated with severe mental illness, particularly schizophrenia. Mechanisms of blunted affect are poorly understood, potentially due to a lack of phenomenological clarity. Here, we examine clinician rated blunted affect and computerized facial metrics derived from ambulatory video assessment using machine learning. With high predictive accuracy (80-82%), we found that head orientation, eye movement, and facets of mouth movement were associated with clinical ratings of blunted affect. Features denoting larger muscle movements were associated with social cognition (R2 = 0.37) and cognition (R2 = 0.40). Findings provide potential insights on psychological and pathophysiological contributors to blunted affect.
Collapse
Affiliation(s)
- Tovah Cowan
- Department of Psychology, Center for Computation and Technology, Louisiana State University, 236 Audubon Hall, Baton Rouge, LA, 70803, USA
| | - Zachary B Rodriguez
- Department of Psychology, Center for Computation and Technology, Louisiana State University, 236 Audubon Hall, Baton Rouge, LA, 70803, USA
| | | | - Ian M Raugh
- Department of Psychology, University of Georgia, Athens, GA, USA
| | - Alex S Cohen
- Department of Psychology, Center for Computation and Technology, Louisiana State University, 236 Audubon Hall, Baton Rouge, LA, 70803, USA.
| |
Collapse
|
6
|
Kiprijanovska I, Stankoski S, Broulidakis MJ, Archer J, Fatoorechi M, Gjoreski M, Nduka C, Gjoreski H. Towards smart glasses for facial expression recognition using OMG and machine learning. Sci Rep 2023; 13:16043. [PMID: 37749176 PMCID: PMC10520037 DOI: 10.1038/s41598-023-43135-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Accepted: 09/20/2023] [Indexed: 09/27/2023] Open
Abstract
This study aimed to evaluate the use of novel optomyography (OMG) based smart glasses, OCOsense, for the monitoring and recognition of facial expressions. Experiments were conducted on data gathered from 27 young adult participants, who performed facial expressions varying in intensity, duration, and head movement. The facial expressions included smiling, frowning, raising the eyebrows, and squeezing the eyes. The statistical analysis demonstrated that: (i) OCO sensors based on the principles of OMG can capture distinct variations in cheek and brow movements with a high degree of accuracy and specificity; (ii) Head movement does not have a significant impact on how well these facial expressions are detected. The collected data were also used to train a machine learning model to recognise the four facial expressions and when the face enters a neutral state. We evaluated this model in conditions intended to simulate real-world use, including variations in expression intensity, head movement and glasses position relative to the face. The model demonstrated an overall accuracy of 93% (0.90 f1-score)-evaluated using a leave-one-subject-out cross-validation technique.
Collapse
Affiliation(s)
| | | | | | | | | | - Martin Gjoreski
- Faculty of Informatics, Università della Svizzera Italiana, 6900, Lugano, Switzerland
| | | | - Hristijan Gjoreski
- Emteq Ltd., Brighton, BN1 9SB, UK
- Faculty of Electrical Engineering and Information Technologies, Ss. Cyril and Methodius University in Skopje, 1000, Skopje, North Macedonia
| |
Collapse
|
7
|
Gupta T, Osborne KJ, Nadig A, Haase CM, Mittal VA. Alterations in facial expressions in individuals at risk for psychosis: a facial electromyography approach using emotionally evocative film clips. Psychol Med 2023; 53:5829-5838. [PMID: 36285533 PMCID: PMC10130238 DOI: 10.1017/s0033291722003087] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
BACKGROUND Negative symptoms such as blunted facial expressivity are characteristic of schizophrenia. However, it is not well-understood if and what abnormalities are present in individuals at clinical high-risk (CHR) for psychosis. METHODS This experimental study employed facial electromyography (left zygomaticus major and left corrugator supercilia) in a sample of CHR individuals (N = 34) and healthy controls (N = 32) to detect alterations in facial expressions in response to emotionally evocative film clips and to determine links with symptoms. RESULTS Findings revealed that the CHR group showed facial blunting manifested in reduced zygomatic activity in response to an excitement (but not amusement, fear, or sadness) film clip compared to controls. Reductions in zygomatic activity in the CHR group emerged in response to the emotionally evocative peak period of the excitement film clip. Lower zygomaticus activity during the excitement clip was related to anxiety while lower rates of change in zygomatic activity during the excitement video clip were related to higher psychosis risk conversion scores. CONCLUSIONS Together, these findings inform vulnerability/disease driving mechanisms and biomarker and treatment development.
Collapse
Affiliation(s)
- Tina Gupta
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, USA
| | - K. Juston Osborne
- Department of Psychology, Northwestern University, Evanston, IL, USA
| | - Ajay Nadig
- Harvard/MIT MD-PhD Program, Harvard Medical School, Boston, MA, 02115
| | - Claudia M. Haase
- Department of Psychology, Northwestern University, Evanston, IL, USA
- School of Education and Social Policy, Northwestern University, Evanston, IL, USA
| | - Vijay A. Mittal
- Department of Psychology, Northwestern University, Evanston, IL, USA
| |
Collapse
|
8
|
Cowan T, Strauss GP, Raugh IM, Le TP, Cohen AS. How do social factors relate to blunted facial affect in schizophrenia? A digital phenotyping study using ambulatory video recordings. J Psychiatr Res 2022; 150:96-104. [PMID: 35366600 PMCID: PMC10036138 DOI: 10.1016/j.jpsychires.2022.03.024] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/17/2021] [Revised: 03/09/2022] [Accepted: 03/21/2022] [Indexed: 01/05/2023]
Abstract
Clinical interviews and laboratory-based emotional induction paradigms provide consistent evidence that facial affect is blunted in many individuals with schizophrenia. Although it is clear that blunted facial affect is not a by-product of diminished emotional experience in schizophrenia, factors contributing to blunted affect remain unclear. The current study used a combination of ambulatory video recordings that were evaluated via computerized facial affect analysis and concurrently completed ecological momentary assessment surveys to assess whether blunted affect reflects insufficient reactivity to affective or contextual factors. Specifically, whether individuals with schizophrenia require more intense affective experiences to produce expression, or whether they are less reactive to social factors (i.e. being in the presence of others, social motivation). Participants included outpatients with schizophrenia (n = 33) and healthy controls (n = 31) who completed six days of study procedures. Multilevel linear models were evaluated using both Null-Hypothesis Statistical Testing and Bayesian analyses. Individuals with schizophrenia displayed comparable expression of positive and negative emotion to controls during daily life, and no evidence was found for a different intensity of experience required for expression in either group. However, social factors differentially influenced facial expression in schizophrenia compared to controls, such that individuals with schizophrenia did not modulate their expressions based on social motivation to the same extent as controls. These findings suggest that social motivation may play an important role in determining when blunting occurs.
Collapse
Affiliation(s)
- Tovah Cowan
- Department of Psychology, Louisiana State University, United States; Center for Computation and Technology, Louisiana State University, United States
| | | | - Ian M Raugh
- Department of Psychology, University of Georgia, United States
| | - Thanh P Le
- Department of Psychology, Louisiana State University, United States
| | - Alex S Cohen
- Department of Psychology, Louisiana State University, United States; Center for Computation and Technology, Louisiana State University, United States.
| |
Collapse
|