1
|
Mielke A, Badihi G, Graham KE, Grund C, Hashimoto C, Piel AK, Safryghin A, Slocombe KE, Stewart F, Wilke C, Zuberbühler K, Hobaiter C. Many morphs: Parsing gesture signals from the noise. Behav Res Methods 2024; 56:6520-6537. [PMID: 38438657 PMCID: PMC11362259 DOI: 10.3758/s13428-024-02368-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/12/2024] [Indexed: 03/06/2024]
Abstract
Parsing signals from noise is a general problem for signallers and recipients, and for researchers studying communicative systems. Substantial efforts have been invested in comparing how other species encode information and meaning, and how signalling is structured. However, research depends on identifying and discriminating signals that represent meaningful units of analysis. Early approaches to defining signal repertoires applied top-down approaches, classifying cases into predefined signal types. Recently, more labour-intensive methods have taken a bottom-up approach describing detailed features of each signal and clustering cases based on patterns of similarity in multi-dimensional feature-space that were previously undetectable. Nevertheless, it remains essential to assess whether the resulting repertoires are composed of relevant units from the perspective of the species using them, and redefining repertoires when additional data become available. In this paper we provide a framework that takes data from the largest set of wild chimpanzee (Pan troglodytes) gestures currently available, splitting gesture types at a fine scale based on modifying features of gesture expression using latent class analysis (a model-based cluster detection algorithm for categorical variables), and then determining whether this splitting process reduces uncertainty about the goal or community of the gesture. Our method allows different features of interest to be incorporated into the splitting process, providing substantial future flexibility across, for example, species, populations, and levels of signal granularity. Doing so, we provide a powerful tool allowing researchers interested in gestural communication to establish repertoires of relevant units for subsequent analyses within and between systems of communication.
Collapse
Affiliation(s)
- Alexander Mielke
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK.
- School of Biological and Behavioural Sciences, Queen Mary University of London, London, UK.
| | - Gal Badihi
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK
| | - Kirsty E Graham
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK
| | - Charlotte Grund
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK
| | - Chie Hashimoto
- Primate Research Institute, Kyoto University, Kyoto, Japan
| | - Alex K Piel
- Department of Anthropology, University College London, London, UK
- Department of Human Origins, Max Planck Institute of Evolutionary Anthropology, Leipzig, Germany
| | - Alexandra Safryghin
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK
| | | | - Fiona Stewart
- Department of Anthropology, University College London, London, UK
- Department of Human Origins, Max Planck Institute of Evolutionary Anthropology, Leipzig, Germany
| | - Claudia Wilke
- Department of Psychology, University of York, York, UK
| | - Klaus Zuberbühler
- Institute of Biology, University of Neuchâtel, Neuchâtel, Switzerland
| | - Catherine Hobaiter
- Wild Minds Lab, School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK
| |
Collapse
|
2
|
Tessier MH, Mazet JP, Gagner E, Marcoux A, Jackson PL. Facial representations of complex affective states combining pain and a negative emotion. Sci Rep 2024; 14:11686. [PMID: 38777852 PMCID: PMC11111784 DOI: 10.1038/s41598-024-62423-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2023] [Accepted: 05/16/2024] [Indexed: 05/25/2024] Open
Abstract
Pain is rarely communicated alone, as it is often accompanied by emotions such as anger or sadness. Communicating these affective states involves shared representations. However, how an individual conceptually represents these combined states must first be tested. The objective of this study was to measure the interaction between pain and negative emotions on two types of facial representations of these states, namely visual (i.e., interactive virtual agents; VAs) and sensorimotor (i.e., one's production of facial configurations). Twenty-eight participants (15 women) read short written scenarios involving only pain or a combined experience of pain and a negative emotion (anger, disgust, fear, or sadness). They produced facial configurations representing these experiences on the faces of the VAs and on their face (own production or imitation of VAs). The results suggest that affective states related to a direct threat to the body (i.e., anger, disgust, and pain) share a similar facial representation, while those that present no immediate danger (i.e., fear and sadness) differ. Although visual and sensorimotor representations of these states provide congruent affective information, they are differently influenced by factors associated with the communication cycle. These findings contribute to our understanding of pain communication in different affective contexts.
Collapse
Affiliation(s)
- Marie-Hélène Tessier
- School of Psychology, Université Laval, Québec City, Canada
- Centre for Interdisciplinary Research in Rehabilitation and Social Integration (Cirris), Québec City, Canada
- CERVO Brain Research Centre, Québec City, Canada
| | - Jean-Philippe Mazet
- Department of Computer Science and Software Engineering, Université Laval, Québec City, Canada
| | - Elliot Gagner
- School of Psychology, Université Laval, Québec City, Canada
- Centre for Interdisciplinary Research in Rehabilitation and Social Integration (Cirris), Québec City, Canada
- CERVO Brain Research Centre, Québec City, Canada
| | - Audrey Marcoux
- School of Psychology, Université Laval, Québec City, Canada
- Centre for Interdisciplinary Research in Rehabilitation and Social Integration (Cirris), Québec City, Canada
- CERVO Brain Research Centre, Québec City, Canada
| | - Philip L Jackson
- School of Psychology, Université Laval, Québec City, Canada.
- Centre for Interdisciplinary Research in Rehabilitation and Social Integration (Cirris), Québec City, Canada.
- CERVO Brain Research Centre, Québec City, Canada.
| |
Collapse
|
3
|
Drummond J, Makdani A, Pawling R, Walker SC. Congenital Anosmia and Facial Emotion Recognition. Physiol Behav 2024; 278:114519. [PMID: 38490365 DOI: 10.1016/j.physbeh.2024.114519] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Revised: 03/11/2024] [Accepted: 03/13/2024] [Indexed: 03/17/2024]
Abstract
Major functions of the olfactory system include guiding ingestion and avoidance of environmental hazards. People with anosmia report reliance on others, for example to check the edibility of food, as their primary coping strategy. Facial expressions are a major source of non-verbal social information that can be used to guide approach and avoidance behaviour. Thus, it is of interest to explore whether a life-long absence of the sense of smell heightens sensitivity to others' facial emotions, particularly those depicting threat. In the present, online study 28 people with congenital anosmia (mean age 43.46) and 24 people reporting no olfactory dysfunction (mean age 42.75) completed a facial emotion recognition task whereby emotionally neutral faces (6 different identities) morphed, over 40 stages, to express one of 5 basic emotions: anger, disgust, fear, happiness, or sadness. Results showed that, while the groups did not differ in their ability to identify the final, full-strength emotional expressions, nor in the accuracy of their first response, the congenital anosmia group successfully identified the emotions at significantly lower intensity (i.e. an earlier stage of the morph) than the control group. Exploratory analysis showed this main effect was primarily driven by an advantage in detecting anger and disgust. These findings indicate the absence of a functioning sense of smell during development leads to compensatory changes in visual, social cognition. Future work should explore the neural and behavioural basis for this advantage.
Collapse
Affiliation(s)
- James Drummond
- Research Centre for Brain & Behaviour, School of Psychology, Faculty of Health, Liverpool John Moores University, Liverpool, UK
| | - Adarsh Makdani
- Research Centre for Brain & Behaviour, School of Psychology, Faculty of Health, Liverpool John Moores University, Liverpool, UK
| | - Ralph Pawling
- Research Centre for Brain & Behaviour, School of Psychology, Faculty of Health, Liverpool John Moores University, Liverpool, UK
| | - Susannah C Walker
- Research Centre for Brain & Behaviour, School of Psychology, Faculty of Health, Liverpool John Moores University, Liverpool, UK.
| |
Collapse
|
4
|
Ikeda S. Interoceptive sensitivity and perception of others' emotions: an investigation based on a two-stage model. Cogn Process 2024; 25:229-239. [PMID: 38383909 DOI: 10.1007/s10339-024-01176-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 01/04/2024] [Indexed: 02/23/2024]
Abstract
Recent research shows that sensitivity to interoceptive sensitivity is associated with a more granular experience of emotions. These studies suggest that individuals sensitive to their interoceptive signals can better perceive somatic physiological changes as compared to their counterparts. Therefore, they discriminate among a wide and subtle range of emotions. Further, the perception of others' emotions could be based on our own emotional experiences. However, whether interoceptive sensitivity is related to the perception of others' emotions remains unclear. Therefore, this study examined the relationship between interoceptive sensitivity and emotional perception. Considering the model that emotion perception comprises two processes, categorization of facial expressions and approach-avoidance responses, this study examined both categorizations of facial expressions and approach-avoidance responses. The results showed no relationship between interoceptive sensitivity and the perception of emotion, which suggests that interoceptive sensitivity is related to the experience of emotion but does not affect the granularity of emotional perception. Future studies should diversely and empirically examine the role of the body in emotional perception from the perspective of interoceptive sensitivity.
Collapse
Affiliation(s)
- Shinnosuke Ikeda
- Human and Social Administration Department, Kanazawa University, Kanazawa University Kakuma-Machi, Kanazawa, Ishikawa Prefecture, 920-1192, Japan.
| |
Collapse
|
5
|
Urtado MB, Rodrigues RD, Fukusima SS. Visual Field Restriction in the Recognition of Basic Facial Expressions: A Combined Eye Tracking and Gaze Contingency Study. Behav Sci (Basel) 2024; 14:355. [PMID: 38785846 PMCID: PMC11117586 DOI: 10.3390/bs14050355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 04/05/2024] [Accepted: 04/18/2024] [Indexed: 05/25/2024] Open
Abstract
Uncertainties and discrepant results in identifying crucial areas for emotional facial expression recognition may stem from the eye tracking data analysis methods used. Many studies employ parameters of analysis that predominantly prioritize the examination of the foveal vision angle, ignoring the potential influences of simultaneous parafoveal and peripheral information. To explore the possible underlying causes of these discrepancies, we investigated the role of the visual field aperture in emotional facial expression recognition with 163 volunteers randomly assigned to three groups: no visual restriction (NVR), parafoveal and foveal vision (PFFV), and foveal vision (FV). Employing eye tracking and gaze contingency, we collected visual inspection and judgment data over 30 frontal face images, equally distributed among five emotions. Raw eye tracking data underwent Eye Movements Metrics and Visualizations (EyeMMV) processing. Accordingly, the visual inspection time, number of fixations, and fixation duration increased with the visual field restriction. Nevertheless, the accuracy showed significant differences among the NVR/FV and PFFV/FV groups, despite there being no difference in NVR/PFFV. The findings underscore the impact of specific visual field areas on facial expression recognition, highlighting the importance of parafoveal vision. The results suggest that eye tracking data analysis methods should incorporate projection angles extending to at least the parafoveal level.
Collapse
Affiliation(s)
- Melina Boratto Urtado
- Faculty of Philosophy, Sciences and Letters at Ribeirão Preto, University of São Paulo, Ribeirão Preto 14040-901, Brazil;
| | | | - Sergio Sheiji Fukusima
- Faculty of Philosophy, Sciences and Letters at Ribeirão Preto, University of São Paulo, Ribeirão Preto 14040-901, Brazil;
| |
Collapse
|
6
|
Malatesta G, D'Anselmo A, Prete G, Lucafò C, Faieta L, Tommasi L. The Predictive Role of the Posterior Cerebellum in the Processing of Dynamic Emotions. CEREBELLUM (LONDON, ENGLAND) 2024; 23:545-553. [PMID: 37285048 PMCID: PMC10951036 DOI: 10.1007/s12311-023-01574-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/29/2023] [Indexed: 06/08/2023]
Abstract
Recent studies have bolstered the important role of the cerebellum in high-level socio-affective functions. In particular, neuroscientific evidence shows that the posterior cerebellum is involved in social cognition and emotion processing, presumably through its involvement in temporal processing and in predicting the outcomes of social sequences. We used cerebellar transcranial random noise stimulation (ctRNS) targeting the posterior cerebellum to affect the performance of 32 healthy participants during an emotion discrimination task, including both static and dynamic facial expressions (i.e., transitioning from a static neutral image to a happy/sad emotion). ctRNS, compared to the sham condition, significantly reduced the participants' accuracy to discriminate static sad facial expressions, but it increased participants' accuracy to discriminate dynamic sad facial expressions. No effects emerged with happy faces. These findings may suggest the existence of two different circuits in the posterior cerebellum for the processing of negative emotional stimuli: a first-time-independent mechanism which can be selectively disrupted by ctRNS, and a second time-dependent mechanism of predictive "sequence detection" which can be selectively enhanced by ctRNS. This latter mechanism might be included among the cerebellar operational models constantly engaged in the rapid adjustment of social predictions based on dynamic behavioral information inherent to others' actions. We speculate that it might be one of the basic principles underlying the understanding of other individuals' social and emotional behaviors during interactions.
Collapse
Affiliation(s)
- Gianluca Malatesta
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy.
| | - Anita D'Anselmo
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy
| | - Giulia Prete
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy
| | - Chiara Lucafò
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy
| | - Letizia Faieta
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy
| | - Luca Tommasi
- Department of Psychological, Health and Territorial Sciences - University "G. d'Annunzio" of Chieti-Pescara, Chieti, Italy
| |
Collapse
|
7
|
Chang CH, Drobotenko N, Ruocco AC, Lee ACH, Nestor A. Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities. Cognition 2024; 245:105724. [PMID: 38266352 DOI: 10.1016/j.cognition.2024.105724] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Revised: 11/09/2023] [Accepted: 01/15/2024] [Indexed: 01/26/2024]
Abstract
Personality traits and affective states are associated with biases in facial emotion perception. However, the precise personality impairments and affective states that underlie these biases remain largely unknown. To investigate how relevant factors influence facial emotion perception and recollection, Experiment 1 employed an image reconstruction approach in which community-dwelling adults (N = 89) rated the similarity of pairs of facial expressions, including those recalled from memory. Subsequently, perception- and memory-based expression representations derived from such ratings were assessed across participants and related to measures of personality impairment, state affect, and visual recognition abilities. Impairment in self-direction and level of positive affect accounted for the largest components of individual variability in perception and memory representations, respectively. Additionally, individual differences in these representations were impacted by face recognition ability. In Experiment 2, adult participants (N = 81) rated facial image reconstructions derived in Experiment 1, revealing that individual variability was associated with specific visual face properties, such as expressiveness, representation accuracy, and positivity/negativity. These findings highlight and clarify the influence of personality, affective state, and recognition abilities on individual differences in the perception and recollection of facial expressions.
Collapse
Affiliation(s)
- Chi-Hsun Chang
- Department of Psychology at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada
| | - Natalia Drobotenko
- Department of Psychology at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada
| | - Anthony C Ruocco
- Department of Psychology at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada; Department of Psychological Clinical Science at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada
| | - Andy C H Lee
- Department of Psychology at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada; Rotman Research Institute, Baycrest Centre, 3560 Bathurst St, North York, Ontario M6A 2E1, Canada
| | - Adrian Nestor
- Department of Psychology at Scarborough, University of Toronto, 1265 Military Trail, Scarborough, Ontario M1C 1A4, Canada.
| |
Collapse
|
8
|
González-Rodríguez A, García-Pérez Á, Godoy-Giménez M, Sayans-Jiménez P, Cañadas F, Estévez AF. The role of the differential outcomes procedure and schizotypy in the recognition of dynamic facial expressions of emotions. Sci Rep 2024; 14:2322. [PMID: 38282111 PMCID: PMC10822869 DOI: 10.1038/s41598-024-52893-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 01/24/2024] [Indexed: 01/30/2024] Open
Abstract
Emotional facial expression recognition is a key ability for adequate social functioning. The current study aims to test if the differential outcomes procedure (DOP) may improve the recognition of dynamic facial expressions of emotions and to further explore whether schizotypal personality traits may have any effect on performance. 183 undergraduate students completed a task where a face morphed from a neutral expression to one of the six basic emotions at full intensity over 10 s. Participants had to press spacebar as soon as they identified the emotion and choose which had appeared. In the first block, participants received no outcomes. In the second block, a group received specific outcomes associated to each emotion (DOP group), while another group received non-differential outcomes after correctly responding (NOP group). Employing generalized linear models (GLMs) and Bayesian inference we estimated different parameters to answer our research goals. Schizotypal personality traits did not seem to affect dynamic emotional facial expression recognition. Participants of the DOP group were less likely to respond incorrectly to faces showing Fear and Surprise at fewer intensity levels. This may suggest that the DOP could lead to better identification of the main features that differentiate each facial expression of emotion.
Collapse
Affiliation(s)
- Antonio González-Rodríguez
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain
- CEINSA Health Research Centre, University of Almería, Almería, Spain
| | - Ángel García-Pérez
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain
- CIBIS Research Centre, University of Almería, Almería, Spain
| | - Marta Godoy-Giménez
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain
- CEINSA Health Research Centre, University of Almería, Almería, Spain
| | - Pablo Sayans-Jiménez
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain
- CEINSA Health Research Centre, University of Almería, Almería, Spain
| | - Fernando Cañadas
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain
- CIBIS Research Centre, University of Almería, Almería, Spain
| | - Angeles F Estévez
- Department of Psychology, University of Almería, Ctra Sacramento S/N, La Cañada de San Urbano, CP: 04120, Almería, Spain.
- CIBIS Research Centre, University of Almería, Almería, Spain.
| |
Collapse
|
9
|
Chen C, Messinger DS, Chen C, Yan H, Duan Y, Ince RAA, Garrod OGB, Schyns PG, Jack RE. Cultural facial expressions dynamically convey emotion category and intensity information. Curr Biol 2024; 34:213-223.e5. [PMID: 38141619 PMCID: PMC10831323 DOI: 10.1016/j.cub.2023.12.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 10/27/2023] [Accepted: 12/01/2023] [Indexed: 12/25/2023]
Abstract
Communicating emotional intensity plays a vital ecological role because it provides valuable information about the nature and likelihood of the sender's behavior.1,2,3 For example, attack often follows signals of intense aggression if receivers fail to retreat.4,5 Humans regularly use facial expressions to communicate such information.6,7,8,9,10,11 Yet how this complex signaling task is achieved remains unknown. We addressed this question using a perception-based, data-driven method to mathematically model the specific facial movements that receivers use to classify the six basic emotions-"happy," "surprise," "fear," "disgust," "anger," and "sad"-and judge their intensity in two distinct cultures (East Asian, Western European; total n = 120). In both cultures, receivers expected facial expressions to dynamically represent emotion category and intensity information over time, using a multi-component compositional signaling structure. Specifically, emotion intensifiers peaked earlier or later than emotion classifiers and represented intensity using amplitude variations. Emotion intensifiers are also more similar across emotions than classifiers are, suggesting a latent broad-plus-specific signaling structure. Cross-cultural analysis further revealed similarities and differences in expectations that could impact cross-cultural communication. Specifically, East Asian and Western European receivers have similar expectations about which facial movements represent high intensity for threat-related emotions, such as "anger," "disgust," and "fear," but differ on those that represent low threat emotions, such as happiness and sadness. Together, our results provide new insights into the intricate processes by which facial expressions can achieve complex dynamic signaling tasks by revealing the rich information embedded in facial expressions.
Collapse
Affiliation(s)
- Chaona Chen
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK.
| | - Daniel S Messinger
- Departments of Psychology, Pediatrics, and Electrical & Computer Engineering, University of Miami, 5665 Ponce De Leon Blvd, Coral Gables, FL 33146, USA
| | - Cheng Chen
- Foreign Language Department, Teaching Centre for General Courses, Chengdu Medical College, 601 Tianhui Street, Chengdu 610083, China
| | - Hongmei Yan
- The MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, North Jianshe Road, Chengdu 611731, China
| | - Yaocong Duan
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK
| | - Robin A A Ince
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK
| | - Oliver G B Garrod
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK
| | - Philippe G Schyns
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK
| | - Rachael E Jack
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, Scotland, UK
| |
Collapse
|
10
|
Richoz AR, Stacchi L, Schaller P, Lao J, Papinutto M, Ticcinelli V, Caldara R. Recognizing facial expressions of emotion amid noise: A dynamic advantage. J Vis 2024; 24:7. [PMID: 38197738 PMCID: PMC10790674 DOI: 10.1167/jov.24.1.7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 11/12/2023] [Indexed: 01/11/2024] Open
Abstract
Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.
Collapse
Affiliation(s)
- Anne-Raphaëlle Richoz
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Lisa Stacchi
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Pauline Schaller
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Junpeng Lao
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Michael Papinutto
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Valentina Ticcinelli
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| |
Collapse
|
11
|
Yu H, Lin C, Sun S, Cao R, Kar K, Wang S. Multimodal investigations of emotional face processing and social trait judgment of faces. Ann N Y Acad Sci 2024; 1531:29-48. [PMID: 37965931 PMCID: PMC10858652 DOI: 10.1111/nyas.15084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2023]
Abstract
Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.
Collapse
Affiliation(s)
- Hongbo Yu
- Department of Psychological & Brain Sciences, University of California Santa Barbara, Santa Barbara, California, USA
| | - Chujun Lin
- Department of Psychology, University of California San Diego, San Diego, California, USA
| | - Sai Sun
- Frontier Research Institute for Interdisciplinary Sciences, Tohoku University, Sendai, Japan
- Research Institute of Electrical Communication, Tohoku University, Sendai, Japan
| | - Runnan Cao
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri, USA
| | - Kohitij Kar
- Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, Canada
| | - Shuo Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri, USA
| |
Collapse
|
12
|
Keating CT, Cook JL. The inside out model of emotion recognition: how the shape of one's internal emotional landscape influences the recognition of others' emotions. Sci Rep 2023; 13:21490. [PMID: 38057460 PMCID: PMC10700588 DOI: 10.1038/s41598-023-48469-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 11/27/2023] [Indexed: 12/08/2023] Open
Abstract
Some people are exceptional at reading emotional expressions, while others struggle. Here we ask whether the way we experience emotion "on the inside" influences the way we expect emotions to be expressed in the "outside world" and subsequently our ability to read others' emotional expressions. Across multiple experiments, incorporating discovery and replication samples, we develop EmoMap (N = 20; N = 271) and ExpressionMap (N = 98; replication N = 193) to map adults' experiences of emotions and visual representations of others' emotions. Some individuals have modular maps, wherein emotional experiences and visual representations are consistent and distinct-anger looks and feels different from happiness, which looks and feels different from sadness. In contrast, others have experiences and representations that are variable and overlapping-anger, happiness, and sadness look and feel similar and are easily confused for one another. Here we illustrate an association between these maps: those with consistent and distinct experiences of emotion also have consistent and distinct visual representations of emotion. Finally (N = 193), we construct the Inside Out Model of Emotion Recognition, which explains 60.8% of the variance in emotion recognition and illuminates multiple pathways to emotion recognition difficulties. These findings have important implications for understanding the emotion recognition difficulties documented in numerous clinical populations.
Collapse
|
13
|
Cheong JH, Jolly E, Xie T, Byrne S, Kenney M, Chang LJ. Py-Feat: Python Facial Expression Analysis Toolbox. AFFECTIVE SCIENCE 2023; 4:781-796. [PMID: 38156250 PMCID: PMC10751270 DOI: 10.1007/s42761-023-00191-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/08/2021] [Accepted: 05/07/2023] [Indexed: 12/30/2023]
Abstract
Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state-of-the-art models require considerable domain expertise that is not traditionally incorporated into social science training programs. Furthermore, there is a notable absence of user-friendly and open-source software that provides a comprehensive set of tools and functions that support facial expression research. In this paper, we introduce Py-Feat, an open-source Python toolbox that provides support for detecting, preprocessing, analyzing, and visualizing facial expression data. Py-Feat makes it easy for domain experts to disseminate and benchmark computer vision models and also for end users to quickly process, analyze, and visualize face expression data. We hope this platform will facilitate increased use of facial expression data in human behavior research. Supplementary Information The online version contains supplementary material available at 10.1007/s42761-023-00191-4.
Collapse
Affiliation(s)
- Jin Hyun Cheong
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
| | - Eshin Jolly
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
| | - Tiankang Xie
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
- Department of Quantitative Biomedical Sciences, Geisel School of Medicine, Dartmouth College, Hanover, NH 03755 USA
| | - Sophie Byrne
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
| | - Matthew Kenney
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
| | - Luke J. Chang
- Computational Social and Affective Neuroscience Laboratory, Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH 03755 USA
- Department of Quantitative Biomedical Sciences, Geisel School of Medicine, Dartmouth College, Hanover, NH 03755 USA
| |
Collapse
|
14
|
Shiota MN, Camras LA, Adolphs R. The Future of Affective Science: Introduction to the Special Issue. AFFECTIVE SCIENCE 2023; 4:429-442. [PMID: 37744969 PMCID: PMC10514001 DOI: 10.1007/s42761-023-00220-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Indexed: 09/26/2023]
Abstract
Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.
Collapse
Affiliation(s)
- Michelle N. Shiota
- Department of Psychology, Arizona State University, P.O. Box 871104, Tempe, AZ 85287-1104 USA
| | - Linda A. Camras
- Department of Psychology, DePaul University, Chicago, IL USA
| | - Ralph Adolphs
- Department of Psychology, Neuroscience, and Biology, California Institute of Technology, Pasadena, CA USA
| |
Collapse
|
15
|
Babenko VV, Yavna DV, Ermakov PN, Anokhina PV. Nonlocal contrast calculated by the second order visual mechanisms and its significance in identifying facial emotions. F1000Res 2023; 10:274. [PMID: 37767361 PMCID: PMC10521119 DOI: 10.12688/f1000research.28396.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/15/2023] [Indexed: 09/29/2023] Open
Abstract
Background: Previously obtained results indicate that faces are / preattentively/ detected in the visual scene very fast, and information on facial expression is rapidly extracted at the lower levels of the visual system. At the same time different facial attributes make different contributions in facial expression recognition. However, it is known, among the preattentive mechanisms there are none that would be selective for certain facial features, such as eyes or mouth. The aim of our study was to identify a candidate for the role of such a mechanism. Our assumption was that the most informative areas of the image are those characterized by spatial heterogeneity, particularly with nonlocal contrast changes. These areas may be identified / in the human visual system/ by the second-order visual / mechanisms/ filters selective to contrast modulations of brightness gradients. Methods: We developed a software program imitating the operation of these / mechanisms/ filters and finding areas of contrast heterogeneity in the image. Using this program, we extracted areas with maximum, minimum and medium contrast modulation amplitudes from the initial face images, then we used these to make three variants of one and the same face. The faces were demonstrated to the observers along with other objects synthesized the same way. The participants had to identify faces and define facial emotional expressions. Results: It was found that the greater is the contrast modulation amplitude of the areas shaping the face, the more precisely the emotion is identified. Conclusions: The results suggest that areas with a greater increase in nonlocal contrast are more informative in facial images, and the second-order visual / mechanisms/ filters can claim the role of /filters/ elements that detect areas of interest, attract visual attention and are windows through which subsequent levels of visual processing receive valuable information.
Collapse
Affiliation(s)
- Vitaly V. Babenko
- Department of Psychophysiology and Clinical Psychology, Academy of Psychology and Education Sciences, Southern Federal University, Rostov-on-Don, Russian Federation
| | - Denis V. Yavna
- Department of Psychophysiology and Clinical Psychology, Academy of Psychology and Education Sciences, Southern Federal University, Rostov-on-Don, Russian Federation
| | - Pavel N. Ermakov
- Department of Psychophysiology and Clinical Psychology, Academy of Psychology and Education Sciences, Southern Federal University, Rostov-on-Don, Russian Federation
| | - Polina V. Anokhina
- Department of Psychophysiology and Clinical Psychology, Academy of Psychology and Education Sciences, Southern Federal University, Rostov-on-Don, Russian Federation
| |
Collapse
|
16
|
Turkstra LS, Hosseini-Moghaddam S, Wohltjen S, Nurre SV, Mutlu B, Duff MC. Facial affect recognition in context in adults with and without TBI. Front Psychol 2023; 14:1111686. [PMID: 37645059 PMCID: PMC10461638 DOI: 10.3389/fpsyg.2023.1111686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 07/31/2023] [Indexed: 08/31/2023] Open
Abstract
Introduction Several studies have reported impaired emotion recognition in adults with traumatic brain injury (TBI), but studies have two major design features that limit application of results to real-world contexts: (1) participants choose from among lists of basic emotions, rather than generating emotion labels, and (2) images are typically presented in isolation rather than in context. To address these limitations, we created an open-labeling task with faces shown alone or in real-life scenes, to more closely approximate how adults with TBI label facial emotions beyond the lab. Methods Participants were 55 adults (29 female) with moderate to severe TBI and 55 uninjured comparison peers, individually matched for race, sex, and age. Participants viewed 60 photographs of faces, either alone or in the pictured person's real-life context, and were asked what that person was feeling. We calculated the percent of responses that were standard forced-choice-task options, and also used sentiment intensity analysis to compare verbal responses between the two groups. We tracked eye movements for a subset of participants, to explore whether gaze duration or number of fixations helped explain any group differences in labels. Results Over 50% of responses in both groups were words other than basic emotions on standard affect tasks, highlighting the importance of eliciting open-ended responses. Valence of labels by participants with TBI was attenuated relative to valence of Comparison group labels, i.e., TBI group responses were less positive to positive images and the same was true for negative images, although the TBI group responses had higher lexical diversity. There were no significant differences in gaze duration or number of fixations between groups. Discussion Results revealed qualitative differences in affect labels between adults with and without TBI that would not have emerged on standard forced-choice tasks. Verbal differences did not appear to be attributable to differences in gaze patterns, leaving open the question of mechanisms of atypical affect processing in adults with TBI.
Collapse
Affiliation(s)
- Lyn S. Turkstra
- Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | | | - Sophie Wohltjen
- Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI, United States
| | - Sara V. Nurre
- American Speech-Language-Hearing Association, Rockville, MD, United States
| | - Bilge Mutlu
- Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI, United States
| | - Melissa C. Duff
- Department of Hearing and Speech Sciences, Vanderbilt University, Nashville, TN, United States
| |
Collapse
|
17
|
Kuipers M, Kappen M, Naber M. How nervous am I? How computer vision succeeds and humans fail in interpreting state anxiety from dynamic facial behaviour. Cogn Emot 2023; 37:1105-1115. [PMID: 37395739 DOI: 10.1080/02699931.2023.2229545] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 04/17/2023] [Accepted: 06/16/2023] [Indexed: 07/04/2023]
Abstract
For human interaction, it is important to understand what emotional state others are in. Especially the observation of faces aids us in putting behaviours into context and gives insight into emotions and mental states of others. Detecting whether someone is nervous, a form of state anxiety, is such an example as it reveals a person's familiarity and contentment with the circumstances. With recent developments in computer vision we developed behavioural nervousness models to show which time-varying facial cues reveal whether someone is nervous in an interview setting. The facial changes, reflecting a state of anxiety, led to more visual exposure and less chemosensory (taste and olfaction) exposure. However, experienced observers had difficulty picking up these changes and failed to detect nervousness levels accurately therewith. This study highlights humans' limited capacity in determining complex emotional states but at the same time provides an automated model that can assist us in achieving fair assessments of so far unexplored emotional states.
Collapse
Affiliation(s)
- Mithras Kuipers
- Experimental Psychology, Helmholtz Institute, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| | - Mitchel Kappen
- Experimental Psychology, Helmholtz Institute, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
- Department of Head and Skin, Ghent University, University Hospital Ghent (UZ Ghent), Ghent, Belgium
| | - Marnix Naber
- Experimental Psychology, Helmholtz Institute, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
18
|
Keating CT, Ichijo E, Cook JL. Autistic adults exhibit highly precise representations of others' emotions but a reduced influence of emotion representations on emotion recognition accuracy. Sci Rep 2023; 13:11875. [PMID: 37481669 PMCID: PMC10363153 DOI: 10.1038/s41598-023-39070-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2023] [Accepted: 07/19/2023] [Indexed: 07/24/2023] Open
Abstract
To date, studies have not yet established the mechanisms underpinning differences in autistic and non-autistic emotion recognition. The current study first investigated whether autistic and non-autistic adults differed in terms of the precision and/or differentiation of their visual emotion representations and their general matching abilities, and second, explored whether differences therein were related to challenges in accurately recognizing emotional expressions. To fulfil these aims, 45 autistic and 45 non-autistic individuals completed three tasks employing dynamic point light displays of emotional facial expressions. We identified that autistic individuals had more precise visual emotion representations than their non-autistic counterparts, however, this did not confer any benefit for their emotion recognition. Whilst for non-autistic people, non-verbal reasoning and the interaction between precision of emotion representations and matching ability predicted emotion recognition, no variables contributed to autistic emotion recognition. These findings raise the possibility that autistic individuals are less guided by their emotion representations, thus lending support to Bayesian accounts of autism.
Collapse
Affiliation(s)
| | - Eri Ichijo
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Jennifer L Cook
- School of Psychology, University of Birmingham, Birmingham, UK
| |
Collapse
|
19
|
Clough S, Morrow E, Mutlu B, Turkstra L, Duff MC. Emotion recognition of faces and emoji in individuals with moderate-severe traumatic brain injury. Brain Inj 2023; 37:596-610. [PMID: 36847497 PMCID: PMC10175112 DOI: 10.1080/02699052.2023.2181401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Revised: 01/11/2023] [Accepted: 02/13/2023] [Indexed: 03/01/2023]
Abstract
BACKGROUND Facial emotion recognition deficits are common after moderate-severe traumatic brain injury (TBI) and linked to poor social outcomes. We examine whether emotion recognition deficits extend to facial expressions depicted by emoji. METHODS Fifty-one individuals with moderate-severe TBI (25 female) and fifty-one neurotypical peers (26 female) viewed photos of human faces and emoji. Participants selected the best-fitting label from a set of basic emotions (anger, disgust, fear, sadness, neutral, surprise, happy) or social emotions (embarrassed, remorseful, anxious, neutral, flirting, confident, proud). RESULTS We analyzed the likelihood of correctly labeling an emotion by group (neurotypical, TBI), stimulus condition (basic faces, basic emoji, social emoji), sex (female, male), and their interactions. Participants with TBI did not significantly differ from neurotypical peers in overall emotion labeling accuracy. Both groups had poorer labeling accuracy for emoji compared to faces. Participants with TBI (but not neurotypical peers) had poorer accuracy for labeling social emotions depicted by emoji compared to basic emotions depicted by emoji. There were no effects of participant sex. DISCUSSION Because emotion representation is more ambiguous in emoji than human faces, studying emoji use and perception in TBI is an important consideration for understanding functional communication and social participation after brain injury.
Collapse
Affiliation(s)
- Sharice Clough
- Department of Hearing and Speech Sciences, Vanderbilt
University Medical Center
| | - Emily Morrow
- Department of Hearing and Speech Sciences, Vanderbilt
University Medical Center
| | - Bilge Mutlu
- Department of Computer Sciences, University of
Wisconsin-Madison
| | - Lyn Turkstra
- School of Rehabilitation Science, McMaster University
| | - Melissa C. Duff
- Department of Hearing and Speech Sciences, Vanderbilt
University Medical Center
| |
Collapse
|
20
|
Ikeda S. Approach-avoidance behavioural patterns towards an affective voice. INTERNATIONAL JOURNAL OF PSYCHOLOGY 2023; 58:164-172. [PMID: 36585807 DOI: 10.1002/ijop.12886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2021] [Accepted: 10/16/2022] [Indexed: 01/01/2023]
Abstract
Facial expressions are evolutionarily acquired social signals that are processed rapidly by the receiver when deciding to either approach or avoid the person sending the signal. The predominance of the approach or avoidance responses has been examined for various facial expressions. Similar to facial expressions, the approach-avoidance response to affective voice may have been acquired evolutionarily. However, studies examining approach-avoidance to an affective voice are limited. This study used the approach-avoidance task framework developed for facial expressions to determine the dominant behavioural responses to affective voices that indicate either happiness or anger. The results show that the approach response was dominant for the happy voice, while the avoidance response was dominant for the angry voice. This tendency was observed even without the presence of any facial stimuli. These results suggest that an affective voice may be an evolutionarily acquired approach-avoidance-inducing social signal, similar to facial expressions.
Collapse
Affiliation(s)
- Shinnosuke Ikeda
- Faculty of Humanities, Kyoto University of Advanced Science, Kyoto, Japan
| |
Collapse
|
21
|
Visual perception of emotion cues in dogs: a critical review of methodologies. Anim Cogn 2023; 26:727-754. [PMID: 36870003 PMCID: PMC10066124 DOI: 10.1007/s10071-023-01762-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 02/07/2023] [Accepted: 02/20/2023] [Indexed: 03/05/2023]
Abstract
Comparative studies of human-dog cognition have grown exponentially since the 2000's, but the focus on how dogs look at us (as well as other dogs) as social partners is a more recent phenomenon despite its importance to human-dog interactions. Here, we briefly summarise the current state of research in visual perception of emotion cues in dogs and why this area is important; we then critically review its most commonly used methods, by discussing conceptual and methodological challenges and associated limitations in depth; finally, we suggest some possible solutions and recommend best practice for future research. Typically, most studies in this field have concentrated on facial emotional cues, with full body information rarely considered. There are many challenges in the way studies are conceptually designed (e.g., use of non-naturalistic stimuli) and the way researchers incorporate biases (e.g., anthropomorphism) into experimental designs, which may lead to problematic conclusions. However, technological and scientific advances offer the opportunity to gather much more valid, objective, and systematic data in this rapidly expanding field of study. Solving conceptual and methodological challenges in the field of emotion perception research in dogs will not only be beneficial in improving research in dog-human interactions, but also within the comparative psychology area, in which dogs are an important model species to study evolutionary processes.
Collapse
|
22
|
Diarra M, Marchitto M, Bressolle MC, Baccino T, Drai-Zerbib V. A narrative review of the interconnection between pilot acute stress, startle, and surprise effects in the aviation context: Contribution of physiological measurements. FRONTIERS IN NEUROERGONOMICS 2023; 4:1059476. [PMID: 38234477 PMCID: PMC10790839 DOI: 10.3389/fnrgo.2023.1059476] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/01/2022] [Accepted: 01/30/2023] [Indexed: 01/19/2024]
Abstract
Aviation remains one of the safest modes of transportation. However, an inappropriate response to an unexpected event can lead to flight incidents and accidents. Among several contributory factors, startle and surprise, which can lead to or exacerbate the pilot's state of stress, are often cited. Unlike stress, which has been the subject of much study in the context of driving and piloting, studies on startle and surprise are less numerous and these concepts are sometimes used interchangeably. Thus, the definitions of stress, startle, and surprise are reviewed, and related differences are put in evidence. Furthermore, it is proposed to distinguish these notions in the evaluation and to add physiological measures to subjective measures in their study. Indeed, Landman's theoretical model makes it possible to show the links between these concepts and studies using physiological parameters show that they would make it possible to disentangle the links between stress, startle and surprise in the context of aviation. Finally, we draw some perspectives to set up further studies focusing specifically on these concepts and their measurement.
Collapse
Affiliation(s)
- Moussa Diarra
- LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, France
| | | | | | - Thierry Baccino
- LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, France
- Université Paris 8, Saint-Denis, France
| | | |
Collapse
|
23
|
Snoek L, Jack RE, Schyns PG, Garrod OG, Mittenbühler M, Chen C, Oosterwijk S, Scholte HS. Testing, explaining, and exploring models of facial expressions of emotions. SCIENCE ADVANCES 2023; 9:eabq8421. [PMID: 36763663 PMCID: PMC9916981 DOI: 10.1126/sciadv.abq8421] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/06/2022] [Accepted: 01/09/2023] [Indexed: 06/18/2023]
Abstract
Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.
Collapse
Affiliation(s)
- Lukas Snoek
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
| | - Rachael E. Jack
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
| | - Philippe G. Schyns
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
| | | | - Maximilian Mittenbühler
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
- Department of Computer Science, University of Tübingen, Tübingen, Germany
| | - Chaona Chen
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
| | - Suzanne Oosterwijk
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - H. Steven Scholte
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| |
Collapse
|
24
|
Mak HW, Lydon-Staley DM, Lunkenheimer E, Lai MHC, Fosco GM. The roles of caregivers and friends in adolescent daily emotion dynamics. SOCIAL DEVELOPMENT 2023; 32:263-282. [PMID: 37664643 PMCID: PMC10470583 DOI: 10.1111/sode.12637] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Accepted: 08/08/2022] [Indexed: 11/30/2022]
Abstract
Daily emotion dynamics provide valuable information about individuals' emotion processes as they go about their lives. Emotion dynamics such as emotion levels (mean), emotion variability (degree of fluctuation), and emotion network density (strength of temporal connections among emotions) are associated with risks for various psychopathology in youth and adults. Prior work has shown that caregivers and friends play crucial socializing roles in adolescent emotional well-being, but less is known about their roles in daily emotion dynamics. This study examined whether caregiver emotion coaching, caregiver-adolescent closeness, and friendship quality were associated with adolescents' emotion levels, emotion variability, and emotion network density. Further, we examined whether caregiver-adolescent closeness moderated the associations between coaching and emotion dynamics. Participants were 150 adolescents (61% girls; Mage = 14.75) and one of their caregivers (95% female; Mage = 43.35) who completed a baseline survey and 21 daily surveys. Results showed that caregiver emotion coaching interacted with caregiver-adolescent closeness in predicting emotion levels and variability. Specifically, when closeness was higher, emotion coaching was significantly associated with lower sadness and anger levels, higher happiness levels, and lower happiness variability. Caregiver emotion coaching, independent of closeness, was also associated with lower anxiety levels, lower sadness variability, and lower emotion network density. Friendship quality was significantly associated with lower levels of sadness, anxiety, and anger, higher levels of happiness, and lower variability in anxiety and anger. These findings suggest that caregivers and friends are central to everyday emotion levels and variability and a more flexible emotion system in adolescents.
Collapse
Affiliation(s)
- Hio Wa Mak
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA 94143, USA
| | - David M. Lydon-Staley
- Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA 19104, USA
- Department of Bioengineering, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104, USA
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Erika Lunkenheimer
- Department of Psychology, The Pennsylvania State University, University Park, PA 16802, USA
| | - Mark H. C. Lai
- Department of Psychology, University of Southern California, Los Angeles, CA, 90089, USA
| | - Gregory M. Fosco
- Department of Human Development and Family Studies, and Edna Bennett Pierce Prevention Research Center, The Pennsylvania State University, University Park, PA 16802, USA
| |
Collapse
|
25
|
Step forward to happiness and back from anger: investigation of approach avoidance responses to facial expressions using a dance pad. CURRENT PSYCHOLOGY 2023. [DOI: 10.1007/s12144-023-04293-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
|
26
|
Zalmenson T, Azriel O, Bar-Haim Y. Enhanced recognition of disgusted expressions occurs in spite of attentional avoidance at encoding. Front Psychol 2023; 13:1063073. [PMID: 36687960 PMCID: PMC9846063 DOI: 10.3389/fpsyg.2022.1063073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Accepted: 12/12/2022] [Indexed: 01/05/2023] Open
Abstract
Introduction Negative emotional content is prioritized in memory. Prioritized attention to negative stimuli has been suggested to mediate this valence-memory association. However, research suggests only a limited role for attention in this observed memory advantage. We tested the role of attention in memory for disgusted facial expressions, a powerful social-emotional stimulus. Methods We measured attention using an incidental, free-viewing encoding task and memory using a surprise memory test for the viewed expressions. Results and Discussion Replicating prior studies, we found increased attentional dwell-time for neutral over disgusted expressions at encoding. However, contrary to the attention-memory link hypothesis, disgusted faces were better remembered than neutral faces. Although dwell-time was found to partially mediate the association between valence and memory, this effect was much weaker than the opposite direct effect. These findings point to independence of memory for disgusted faces from attention during encoding.
Collapse
|
27
|
Tonnaer F, van Zutphen L, Raine A, Cima M. Amygdala connectivity and aggression. HANDBOOK OF CLINICAL NEUROLOGY 2023; 197:87-106. [PMID: 37633721 DOI: 10.1016/b978-0-12-821375-9.00002-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/28/2023]
Abstract
Neurobiological models propose that reactive aggression is predicated on impairments in amygdala-prefrontal connectivity that subserves moral decision-making and emotion regulation. The amygdala is a key component within this neural network that modulates reactive aggression. We provide a review of amygdala dysfunctional brain networks leading to reactive aggressive behavior. We elaborate on key concepts, focusing on moral decision-making and emotion regulation in a developmental context, and brain network connectivity factors relating to amygdala (dys)function-factors which we suggest predispose to reactive aggression. We additionally discuss insights into the latest treatment interventions, providing the utilization of the scientific findings for practice.
Collapse
Affiliation(s)
- Franca Tonnaer
- Department of Research, Ventio Crime Prevention Science Institute, Rijckholt, The Netherlands
| | - Linda van Zutphen
- Department of Conditions for LifeLong Learning, Educational Sciences, Open University, Heerlen, The Netherlands
| | - Adrian Raine
- Department of Criminology, Richard Perry University, Berkeley, CA, United States; Department of Psychiatry, University of Pennsylvania, Philadelphia, PA, United States
| | - Maaike Cima
- Department of Research, Ventio Crime Prevention Science Institute, Rijckholt, The Netherlands; Department of Developmental Psychopathology, Behavioural Science Institute, Radboud University, Nijmegen, The Netherlands; Department of Research, VIGO Groep, Nijmegen, The Netherlands.
| |
Collapse
|
28
|
Smith RA, Cross ES. The McNorm library: creating and validating a new library of emotionally expressive whole body dance movements. PSYCHOLOGICAL RESEARCH 2023; 87:484-508. [PMID: 35385989 PMCID: PMC8985749 DOI: 10.1007/s00426-022-01669-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2021] [Accepted: 02/23/2022] [Indexed: 11/28/2022]
Abstract
The ability to exchange affective cues with others plays a key role in our ability to create and maintain meaningful social relationships. We express our emotions through a variety of socially salient cues, including facial expressions, the voice, and body movement. While significant advances have been made in our understanding of verbal and facial communication, to date, understanding of the role played by human body movement in our social interactions remains incomplete. To this end, here we describe the creation and validation of a new set of emotionally expressive whole-body dance movement stimuli, named the Motion Capture Norming (McNorm) Library, which was designed to reconcile a number of limitations associated with previous movement stimuli. This library comprises a series of point-light representations of a dancer's movements, which were performed to communicate to observers neutrality, happiness, sadness, anger, and fear. Based on results from two validation experiments, participants could reliably discriminate the intended emotion expressed in the clips in this stimulus set, with accuracy rates up to 60% (chance = 20%). We further explored the impact of dance experience and trait empathy on emotion recognition and found that neither significantly impacted emotion discrimination. As all materials for presenting and analysing this movement library are openly available, we hope this resource will aid other researchers in further exploration of affective communication expressed by human bodily movement.
Collapse
Affiliation(s)
- Rebecca A. Smith
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, Scotland
| | - Emily S. Cross
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, Scotland ,Department of Cognitive Science, Macquarie University, Sydney, Australia
| |
Collapse
|
29
|
FEDA: Fine-grained emotion difference analysis for facial expression recognition. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
30
|
Huang J, Zhao Y, Qu W, Tian Z, Tan Y, Wang Z, Tan S. Automatic recognition of schizophrenia from facial videos using 3D convolutional neural network. Asian J Psychiatr 2022; 77:103263. [PMID: 36152565 DOI: 10.1016/j.ajp.2022.103263] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Revised: 08/22/2022] [Accepted: 09/14/2022] [Indexed: 11/17/2022]
Abstract
Schizophrenia affects patients and their families and society because of chronic impairments in cognition, behavior, and emotion. However, its clinical diagnosis mainly depends on the clinicians' knowledge of the patients' symptoms. Other auxiliary diagnostic methods such as MRI and EEG are cumbersome and time-consuming. Recently, the convolutional neural network (CNN) has been applied to the auxiliary diagnosis of psychiatry. Hence, in this study, a method based on deep learning and facial videos is proposed for the rapid detection of schizophrenia. Herein, 125 videos from 125 schizophrenic patients and 75 videos from 75 healthy controls based on emotional stimulation tasks were obtained. The video preprocessing included the experiment clips extraction, face detection, facial region cropping, resizing to 500 × 500 pixel size, and uniform sampling of 100 frames. The preprocessed facial videos were used to train the Resnet18_3D. We utilized ten-fold cross-validation, and held-out testing set to evaluate the model with the accuracy, the precision, the sensitivity, the specificity, the balanced accuracy, and the AUC. The Resnet18_3D trained on Film_order achieved the best performance with accuracy, sensitivity, specificity, balanced accuracy, and AUC of 89.00%, 96.80%, 76.00%, 86.40% and 0.9397. The neural network model indeed recognizes healthy controls and schizophrenic patients through the changes in the area of the face. The results show that facial video under emotional stimulation can be used to classify schizophrenic patients and help clinicians with diagnosis in the clinical environment. Among the different types of stimuli, the video stimuli with fixed emotional order showed the best classification performance.
Collapse
Affiliation(s)
- Jie Huang
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Yanli Zhao
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Wei Qu
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Zhanxiao Tian
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Yunlong Tan
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Zhiren Wang
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China
| | - Shuping Tan
- Beijing HuiLongGuan Hospital, Peking University HuiLongGuan Clinical Medical School, Beijing, 100096, China.
| |
Collapse
|
31
|
Díaz-Agea JL, Pujalte-Jesús MJ, Arizo-Luque V, García-Méndez JA, López-Chicheri-García I, Rojo-Rojo A. How Are You Feeling? Interpretation of Emotions through Facial Expressions of People Wearing Different Personal Protective Equipment: An Observational Study. NURSING REPORTS 2022; 12:758-774. [PMID: 36278768 PMCID: PMC9590080 DOI: 10.3390/nursrep12040075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2022] [Revised: 10/10/2022] [Accepted: 10/13/2022] [Indexed: 11/05/2022] Open
Abstract
(1) Background: The perception of others’ emotions based on non-verbal cues, such as facial expressions, is fundamental for interpersonal communication and mutual support. Using personal protection equipment (PPE) in a work environment during the SAR-CoV-2 pandemic challenged health professionals’ ability to recognise emotions and expressions while wearing PPE. The working hypothesis of this study was that the increased limitation of facial visibility, due to the use of a personal protective device, would interfere with the perception of basic emotions in the participants. (2) Methods: Through a cross-sectional descriptive study, the present research aimed to analyse the identification of four basic emotions (happiness; sadness; fear/surprise; and disgust/anger) through three types of PPE (FFP2 respirator, protective overall and powered air-purifying respirator (PAPR)), by using 32 photographs. The study was conducted using volunteer participants who met the inclusion criteria (individuals older than 13 without cognitive limitations). Participants had to recognise the emotions of actors in photographs that were randomly displayed in an online form. (3) Results: In general, the 690 participants better recognised happiness and fear, independently of the PPE utilised. Women could better identify different emotions, along with university graduates and young and middle-aged adults. Emotional identification was at its worst when the participants wore protective overalls (5.42 ± 1.22), followed by the PAPR (5.83 ± 1.38); the best scores were obtained using the FFP2 masks (6.57 ± 1.20). Sadness was the least recognised emotion, regardless of age. (4) Conclusions: The personal protective devices interfere in the recognition of emotions, with the protective overalls having the greatest impact, and the FFP2 mask the least. The emotions that were best recognised were happiness and fear/surprise, while the least recognised emotion was sadness. Women were better at identifying emotions, as well as participants with higher education, and young and middle-aged adults.
Collapse
|
32
|
Bylianto LO, Chan KQ. Face masks inhibit facial cues for approachability and trustworthiness: an eyetracking study. CURRENT PSYCHOLOGY 2022; 42:1-12. [PMID: 36217421 PMCID: PMC9535231 DOI: 10.1007/s12144-022-03705-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/24/2022] [Indexed: 11/03/2022]
Abstract
Wearing face masks during the Covid-19 pandemic has undeniable benefits from our health perspective. However, the interpersonal costs on social interactions may have been underappreciated. Because masks obscure critical facial regions signaling approach/avoidance intent and social trust, this implies that facial inference of approachability and trustworthiness may be severely discounted. Here, in our eyetracking experiment, we show that people judged masked faces as less approachable and trustworthy. Further analyses showed that the attention directed towards the eye region relative to the mouth region mediated the effect on approachability, but not on trustworthiness. This is because for masked faces, with the mouth region obscured, visual attention is then automatically diverted away from the mouth and towards the eye region, which is an undiagnostic cue for judging a target's approachability. Together, these findings support that mask-wearing inhibits the critical facial cues needed for social judgements. Supplementary Information The online version contains supplementary material available at 10.1007/s12144-022-03705-8.
Collapse
|
33
|
Kuruvayil S, Palaniswamy S. Emotion recognition from facial images with simultaneous occlusion, pose and illumination variations using meta-learning. JOURNAL OF KING SAUD UNIVERSITY - COMPUTER AND INFORMATION SCIENCES 2022. [DOI: 10.1016/j.jksuci.2021.06.012] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
34
|
Bour C, Ahne A, Aguayo G, Fischer A, Marcic D, Kayser P, Fagherazzi G. Global diabetes burden: analysis of regional differences to improve diabetes care. BMJ Open Diabetes Res Care 2022; 10:10/5/e003040. [PMID: 36307139 PMCID: PMC9621169 DOI: 10.1136/bmjdrc-2022-003040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Accepted: 10/14/2022] [Indexed: 11/07/2022] Open
Abstract
INTRODUCTION The current evaluation processes of the burden of diabetes are incomplete and subject to bias. This study aimed to identify regional differences in the diabetes burden on a universal level from the perspective of people with diabetes. RESEARCH DESIGN AND METHODS We developed a worldwide online diabetes observatory based on 34 million diabetes-related tweets from 172 countries covering 41 languages, spanning from 2017 to 2021. After translating all tweets to English, we used machine learning algorithms to remove institutional tweets and jokes, geolocate users, identify topics of interest and quantify associated sentiments and emotions across the seven World Bank regions. RESULTS We identified four topics of interest for people with diabetes (PWD) in the Middle East and North Africa and another 18 topics in North America. Topics related to glycemic control and food are shared among six regions of the world. These topics were mainly associated with sadness (35% and 39% on average compared with levels of sadness in other topics). We also revealed several region-specific concerns (eg, insulin pricing in North America or the burden of daily diabetes management in Europe and Central Asia). CONCLUSIONS The needs and concerns of PWD vary significantly worldwide, and the burden of diabetes is perceived differently. Our results will support better integration of these regional differences into diabetes programs to improve patient-centric diabetes research and care, focused on the most relevant concerns to enhance personalized medicine and self-management of PWD.
Collapse
Affiliation(s)
- Charline Bour
- Department of Precision Health, Deep Digital Phenotyping Research Unit, Luxembourg Institute of Health, Strassen, Luxembourg
- Faculty of Science, Technology and Medicine, University of Luxembourg, Esch-sur-Alzette, Luxembourg
| | - Adrian Ahne
- Center for Research in Epidemiology and Population Health (CESP), INSERM, Villejuif (Paris), Île-de-France, France
| | - Gloria Aguayo
- Department of Precision Health, Deep Digital Phenotyping Research Unit, Luxembourg Institute of Health, Strassen, Luxembourg
| | - Aurélie Fischer
- Department of Precision Health, Deep Digital Phenotyping Research Unit, Luxembourg Institute of Health, Strassen, Luxembourg
| | - David Marcic
- Department of Precision Health, Data Integration and Analysis Unit, Luxembourg Institute of Health, Strassen, Luxembourg
| | - Philippe Kayser
- Department of Precision Health, Data Integration and Analysis Unit, Luxembourg Institute of Health, Strassen, Luxembourg
| | - Guy Fagherazzi
- Department of Precision Health, Deep Digital Phenotyping Research Unit, Luxembourg Institute of Health, Strassen, Luxembourg
| |
Collapse
|
35
|
Hsu CT, Sato W, Kochiyama T, Nakai R, Asano K, Abe N, Yoshikawa S. Enhanced Mirror Neuron Network Activity and Effective Connectivity during Live Interaction Among Female Subjects. Neuroimage 2022; 263:119655. [PMID: 36182055 DOI: 10.1016/j.neuroimage.2022.119655] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2022] [Revised: 09/26/2022] [Accepted: 09/27/2022] [Indexed: 11/24/2022] Open
Abstract
Facial expressions are indispensable in daily human communication. Previous neuroimaging studies investigating facial expression processing have presented pre-recorded stimuli and lacked live face-to-face interaction. Our paradigm alternated between presentations of real-time model performance and pre-recorded videos of dynamic facial expressions to participants. Simultaneous functional magnetic resonance imaging (fMRI) and facial electromyography activity recordings, as well as post-scan valence and arousal ratings were acquired from 44 female participants. Live facial expressions enhanced the subjective valence and arousal ratings as well as facial muscular responses. Live performances showed greater engagement of the right posterior superior temporal sulcus (pSTS), right inferior frontal gyrus (IFG), right amygdala and right fusiform gyrus, and modulated the effective connectivity within the right mirror neuron system (IFG, pSTS, and right inferior parietal lobule). A support vector machine algorithm could classify multivoxel activation patterns in brain regions involved in dynamic facial expression processing in the mentalizing networks (anterior and posterior cingulate cortex). These results indicate that live social interaction modulates the activity and connectivity of the right mirror neuron system and enhances spontaneous mimicry, further facilitating emotional contagion.
Collapse
Affiliation(s)
- Chun-Ting Hsu
- Psychological Process Research Team, Guardian Robot Project, RIKEN, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan..
| | - Wataru Sato
- Psychological Process Research Team, Guardian Robot Project, RIKEN, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan..
| | - Takanori Kochiyama
- Brain Activity Imaging Center, ATR- Promotions, Inc., 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
| | - Ryusuke Nakai
- Institute for the Future of Human Society, Kyoto University, 46 Yoshidashimoadachi-cho, Sakyo-ku, Kyoto, 606-8501 Japan
| | - Kohei Asano
- Institute for the Future of Human Society, Kyoto University, 46 Yoshidashimoadachi-cho, Sakyo-ku, Kyoto, 606-8501 Japan; Department of Children Education, Osaka University of Comprehensive Children Education, 6-chome-4-26 Yuzato, Higashisumiyoshi Ward, Osaka, 546-0013, Japan
| | - Nobuhito Abe
- Institute for the Future of Human Society, Kyoto University, 46 Yoshidashimoadachi-cho, Sakyo-ku, Kyoto, 606-8501 Japan
| | - Sakiko Yoshikawa
- Institute of Philosophy and Human Values, Kyoto University of the Arts, 2-116 Uryuyama Kitashirakawa, Sakyo, Kyoto, Kyoto 606-8271, Japan
| |
Collapse
|
36
|
Lau WK, Sauter M, Huckauf A. Small Pupils Lead to Lower Judgements of a Person’s Characteristics for Exaggerated, but Not for Realistic Pupils. Behav Sci (Basel) 2022; 12:bs12080283. [PMID: 36004854 PMCID: PMC9405288 DOI: 10.3390/bs12080283] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2022] [Revised: 08/03/2022] [Accepted: 08/10/2022] [Indexed: 11/16/2022] Open
Abstract
Our eyes convey information about a person. The pupils may provide information regarding our emotional states when presented along with different emotional expressions. We examined the effects of pupil size and vergence on inferring other people’s characteristics in neutral expression eyes. Pupil sizes were manipulated by overlaying black disks onto the pupils of the original eye images. The disk area was then changed to create small, medium, and large pupils. Vergence was simulated by shifting the medium-sized disks nasally in one eye. Pupil sizes were exaggerated for Experiment 1 and followed values from the literature for Experiment 2. The first Purkinje image from the eye photos in Experiment 2 was kept to preserve image realism. The characteristics measured were sex, age, attractiveness, trustworthiness, intelligence, valence, and arousal. Participants completed one of two online experiments and rated eight eye pictures with differently sized pupils and with vergence eyes. Both experiments were identical except for the stimuli designs. Results from Experiment 1 revealed rating differences between pupil sizes for all characteristics except sex, age, and arousal. Specifically, eyes with extremely small pupil sizes and artificial vergence received the lowest ratings compared to medium and large pupil sizes. Results from Experiment 2 only indicated weak effects of pupil size and vergence, particularly for intelligence ratings. We conclude that the pupils can influence how characteristics of another person are perceived and may be regarded as important social signals in subconscious social interaction processes. However, the effects may be rather small for neutral expressions.
Collapse
|
37
|
del Valle-Canencia M, Moreno Martínez C, Rodríguez-Jiménez RM, Corrales-Paredes A. The emotions effect on a virtual characters design–A student perspective analysis. FRONTIERS IN COMPUTER SCIENCE 2022. [DOI: 10.3389/fcomp.2022.892597] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
Interaction between people and virtual characters through digital and electronic devices is a reality. In this context, the design of virtual characters must incorporate emotional expression at a nonverbal level looking for effective communication with the user. This exploratory study investigates the design features of an avatar functioning as a virtual assistant in educational contexts. From a multidisciplinary approach, the user's research was elaborated by a semi-open questionnaire of self-perception of emotional characteristics: likeability, attractiveness, and applicability of a set of six 2D and 3D characters. The results extracted from a sample of 69 university students provide a relevant information on design features and open new lines for future research. Aspects such as Ekman's basic emotion discrimination and the design of facial expression are analyzed. The incorporation of other body parts, their spatial orientation and contextual elements, seems to contribute to effective emotional communication. The results also highlight how the design of a virtual character should take into consideration the complexity involved in facial gestures and changes in relation to the vertical axis and planes of movement. Finally, this article discusses the complexity involved in expressing a given emotion in a virtual character.
Collapse
|
38
|
Namba S, Nakamura K, Watanabe K. The spatio-temporal features of perceived-as-genuine and deliberate expressions. PLoS One 2022; 17:e0271047. [PMID: 35839208 PMCID: PMC9286247 DOI: 10.1371/journal.pone.0271047] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2021] [Accepted: 06/22/2022] [Indexed: 11/24/2022] Open
Abstract
Reading the genuineness of facial expressions is important for increasing the credibility of information conveyed by faces. However, it remains unclear which spatio-temporal characteristics of facial movements serve as critical cues to the perceived genuineness of facial expressions. This study focused on observable spatio-temporal differences between perceived-as-genuine and deliberate expressions of happiness and anger expressions. In this experiment, 89 Japanese participants were asked to judge the perceived genuineness of faces in videos showing happiness or anger expressions. To identify diagnostic facial cues to the perceived genuineness of the facial expressions, we analyzed a total of 128 face videos using an automated facial action detection system; thereby, moment-to-moment activations in facial action units were annotated, and nonnegative matrix factorization extracted sparse and meaningful components from all action units data. The results showed that genuineness judgments reduced when more spatial patterns were observed in facial expressions. As for the temporal features, the perceived-as-deliberate expressions of happiness generally had faster onsets to the peak than the perceived-as-genuine expressions of happiness. Moreover, opening the mouth negatively contributed to the perceived-as-genuine expressions, irrespective of the type of facial expressions. These findings provide the first evidence for dynamic facial cues to the perceived genuineness of happiness and anger expressions.
Collapse
Affiliation(s)
- Shushi Namba
- Psychological Process Research Team, Guardian Robot Project, RIKEN, Kyoto, Japan
| | - Koyo Nakamura
- Faculty of Psychology, Department of Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna, Austria
- Japan Society for the Promotion of Science, Tokyo, Japan
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
| | - Katsumi Watanabe
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
| |
Collapse
|
39
|
Ge Y, Su R, Liang Z, Luo J, Tian S, Shen X, Wu H, Liu C. Transcranial Direct Current Stimulation Over the Right Temporal Parietal Junction Facilitates Spontaneous Micro-Expression Recognition. Front Hum Neurosci 2022; 16:933831. [PMID: 35874155 PMCID: PMC9305610 DOI: 10.3389/fnhum.2022.933831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2022] [Accepted: 06/21/2022] [Indexed: 11/19/2022] Open
Abstract
Micro-expressions are fleeting and subtle emotional expressions. As they are spontaneous and uncontrollable by one's mind, micro-expressions are considered an indicator of genuine emotions. Their accurate recognition and interpretation promote interpersonal interaction and social communication. Therefore, enhancing the ability to recognize micro-expressions has captured much attention. In the current study, we investigated the effects of training on micro-expression recognition with a Chinese version of the Micro-Expression Training Tool (METT). Our goal was to confirm whether the recognition accuracy of spontaneous micro-expressions could be improved through training and brain stimulation. Since the right temporal parietal junction (rTPJ) has been shown to be involved in the explicit process of facial emotion recognition, we hypothesized that the rTPJ would play a role in facilitating the recognition of micro-expressions. The results showed that anodal transcranial direct-current stimulation (tDCS) of the rTPJ indeed improved the recognition of spontaneous micro-expressions, especially for those associated with fear. The improved accuracy of recognizing fear spontaneous micro-expressions was positively correlated with personal distress in the anodal group but not in the sham group. Our study supports that the combined use of tDCS and METT can be a viable way to train and enhance micro-expression recognition.
Collapse
Affiliation(s)
- Yue Ge
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Center for Collaboration and Innovation in Brain and Learning Sciences, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
- Beijing Institute of Biomedicine, Beijing, China
| | - Rui Su
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Center for Collaboration and Innovation in Brain and Learning Sciences, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
| | - Zilu Liang
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Center for Collaboration and Innovation in Brain and Learning Sciences, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
| | - Jing Luo
- Beijing Institute of Biomedicine, Beijing, China
| | - Suizi Tian
- School of Psychology, Beijing Normal University, Beijing, China
| | - Xunbing Shen
- College of Humanities, Jiangxi University of Chinese Medicine, Nanchang, China
| | - Haiyan Wu
- Centre for Cognitive and Brain Sciences and Department of Psychology, University of Macau, Taipa, China
| | - Chao Liu
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
- Center for Collaboration and Innovation in Brain and Learning Sciences, Beijing Normal University, Beijing, China
- Beijing Key Laboratory of Brain Imaging and Connectomics, Beijing Normal University, Beijing, China
| |
Collapse
|
40
|
Kato M, Sato H, Mizokami Y. Effect of skin colors due to hemoglobin or melanin modulation on facial expression recognition. Vision Res 2022; 196:108048. [DOI: 10.1016/j.visres.2022.108048] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2021] [Revised: 03/15/2022] [Accepted: 03/23/2022] [Indexed: 11/30/2022]
|
41
|
Namba S, Sato W, Matsui H. Spatio-Temporal Properties of Amused, Embarrassed, and Pained Smiles. JOURNAL OF NONVERBAL BEHAVIOR 2022. [DOI: 10.1007/s10919-022-00404-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
Abstract
AbstractSmiles are universal but nuanced facial expressions that are most frequently used in face-to-face communications, typically indicating amusement but sometimes conveying negative emotions such as embarrassment and pain. Although previous studies have suggested that spatial and temporal properties could differ among these various types of smiles, no study has thoroughly analyzed these properties. This study aimed to clarify the spatiotemporal properties of smiles conveying amusement, embarrassment, and pain using a spontaneous facial behavior database. The results regarding spatial patterns revealed that pained smiles showed less eye constriction and more overall facial tension than amused smiles; no spatial differences were identified between embarrassed and amused smiles. Regarding temporal properties, embarrassed and pained smiles remained in a state of higher facial tension than amused smiles. Moreover, embarrassed smiles showed a more gradual change from tension states to the smile state than amused smiles, and pained smiles had lower probabilities of staying in or transitioning to the smile state compared to amused smiles. By comparing the spatiotemporal properties of these three smile types, this study revealed that the probability of transitioning between discrete states could help distinguish amused, embarrassed, and pained smiles.
Collapse
|
42
|
Li S, Ding R, Zhao D, Zhou X, Zhan B, Luo W. Processing of emotions expressed through eye regions attenuates attentional blink. Int J Psychophysiol 2022; 182:1-11. [DOI: 10.1016/j.ijpsycho.2022.07.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Revised: 07/13/2022] [Accepted: 07/27/2022] [Indexed: 10/16/2022]
|
43
|
Faustmann LL, Eckhardt L, Hamann PS, Altgassen M. The Effects of Separate Facial Areas on Emotion Recognition in Different Adult Age Groups: A Laboratory and a Naturalistic Study. Front Psychol 2022; 13:859464. [PMID: 35846682 PMCID: PMC9281501 DOI: 10.3389/fpsyg.2022.859464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Accepted: 06/07/2022] [Indexed: 11/13/2022] Open
Abstract
The identification of facial expressions is critical for social interaction. The ability to recognize facial emotional expressions declines with age. These age effects have been associated with differential age-related looking patterns. The present research project set out to systematically test the role of specific facial areas for emotion recognition across the adult lifespan. Study 1 investigated the impact of displaying only separate facial areas versus the full face on emotion recognition in 62 younger (20-24 years) and 65 middle-aged adults (40-65 years). Study 2 examined if wearing face masks differentially compromises younger (18-33 years, N = 71) versus middle-aged to older adults' (51-83 years, N = 73) ability to identify different emotional expressions. Results of Study 1 suggested no general decrease in emotion recognition across the lifespan; instead, age-related performance seems to depend on the specific emotion and presented face area. Similarly, Study 2 observed only deficits in the identification of angry, fearful, and neutral expressions in older adults, but no age-related differences with regards to happy, sad, and disgusted expressions. Overall, face masks reduced participants' emotion recognition; however, there were no differential age effects. Results are discussed in light of current models of age-related changes in emotion recognition.
Collapse
Affiliation(s)
| | | | | | - Mareike Altgassen
- Department of Psychology, Johannes Gutenberg University Mainz, Mainz, Germany
| |
Collapse
|
44
|
Measuring ethical behavior with AI and natural language processing to assess business success. Sci Rep 2022; 12:10228. [PMID: 35715458 PMCID: PMC9205897 DOI: 10.1038/s41598-022-14101-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2022] [Accepted: 06/01/2022] [Indexed: 11/13/2022] Open
Abstract
Everybody claims to be ethical. However, there is a huge difference between declaring ethical behavior and living up to high ethical standards. In this paper, we demonstrate that “hidden honest signals” in the language and the use of “small words” can show true moral values and behavior of individuals and organizations and that this ethical behavior is correlated to real-world success; however not always in the direction we might expect. Leveraging the latest advances of AI in natural language processing (NLP), we construct three different “tribes” of ethical, moral, and non-ethical people, based on Twitter feeds of people of known high and low ethics and morals: fair and modest collaborators codified as ethical “bees”; hard-working competitive workers as moral “ants”; and selfish, arrogant people as non-ethical “leeches”. Results from three studies involving a total of 49 workgroups and 281 individuals within three different industries (healthcare, business consulting, and higher education) confirm the validity of our model. Associating membership in ethical or unethical tribes with performance, we find that being ethical correlates positively or negatively with success depending on the context.
Collapse
|
45
|
Synthesizing a Talking Child Avatar to Train Interviewers Working with Maltreated Children. BIG DATA AND COGNITIVE COMPUTING 2022. [DOI: 10.3390/bdcc6020062] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
When responding to allegations of child sexual, physical, and psychological abuse, Child Protection Service (CPS) workers and police personnel need to elicit detailed and accurate accounts of the abuse to assist in decision-making and prosecution. Current research emphasizes the importance of the interviewer’s ability to follow empirically based guidelines. In doing so, it is essential to implement economical and scientific training courses for interviewers. Due to recent advances in artificial intelligence, we propose to generate a realistic and interactive child avatar, aiming to mimic a child. Our ongoing research involves the integration and interaction of different components with each other, including how to handle the language, auditory, emotional, and visual components of the avatar. This paper presents three subjective studies that investigate and compare various state-of-the-art methods for implementing multiple aspects of the child avatar. The first user study evaluates the whole system and shows that the system is well received by the expert and highlights the importance of its realism. The second user study investigates the emotional component and how it can be integrated with video and audio, and the third user study investigates realism in the auditory and visual components of the avatar created by different methods. The insights and feedback from these studies have contributed to the refined and improved architecture of the child avatar system which we present here.
Collapse
|
46
|
Namba S, Sato W, Nakamura K, Watanabe K. Computational Process of Sharing Emotion: An Authentic Information Perspective. Front Psychol 2022; 13:849499. [PMID: 35645906 PMCID: PMC9134197 DOI: 10.3389/fpsyg.2022.849499] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2022] [Accepted: 04/26/2022] [Indexed: 11/28/2022] Open
Abstract
Although results of many psychology studies have shown that sharing emotion achieves dyadic interaction, no report has explained a study of the transmission of authentic information from emotional expressions that can strengthen perceivers. For this study, we used computational modeling, which is a multinomial processing tree, for formal quantification of the process of sharing emotion that emphasizes the perception of authentic information for expressers’ feeling states from facial expressions. Results indicated that the ability to perceive authentic information of feeling states from a happy expression has a higher probability than the probability of judging authentic information from anger expressions. Next, happy facial expressions can activate both emotional elicitation and sharing emotion in perceivers, where emotional elicitation alone is working rather than sharing emotion for angry facial expressions. Third, parameters to detect anger experiences were found to be correlated positively with those of happiness. No robust correlation was found between the parameters extracted from this experiment task and questionnaire-measured emotional contagion, empathy, and social anxiety. Results of this study revealed the possibility that a new computational approach contributes to description of emotion sharing processes.
Collapse
Affiliation(s)
- Shushi Namba
- Psychological Process Research Team, Guardian Robot Project, RIKEN, Kyoto, Japan
- *Correspondence: Shushi Namba,
| | - Wataru Sato
- Psychological Process Research Team, Guardian Robot Project, RIKEN, Kyoto, Japan
| | - Koyo Nakamura
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
- Japan Society for the Promotion of Science, Tokyo, Japan
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
| | - Katsumi Watanabe
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
- Faculty of Arts, Design and Architecture, University of New South Wales, Sydney, NSW, Australia
| |
Collapse
|
47
|
Features and Extra-Striate Body Area Representations of Diagnostic Body Parts in Anger and Fear Perception. Brain Sci 2022; 12:brainsci12040466. [PMID: 35447997 PMCID: PMC9028525 DOI: 10.3390/brainsci12040466] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 03/19/2022] [Accepted: 03/29/2022] [Indexed: 02/04/2023] Open
Abstract
Social species perceive emotion via extracting diagnostic features of body movements. Although extensive studies have contributed to knowledge on how the entire body is used as context for decoding bodily expression, we know little about whether specific body parts (e.g., arms and legs) transmit enough information for body understanding. In this study, we performed behavioral experiments using the Bubbles paradigm on static body images to directly explore diagnostic body parts for categorizing angry, fearful and neutral expressions. Results showed that subjects recognized emotional bodies through diagnostic features from the torso with arms. We then conducted a follow-up functional magnetic resonance imaging (fMRI) experiment on body part images to examine whether diagnostic parts modulated body-related brain activity and corresponding neural representations. We found greater activations of the extra-striate body area (EBA) in response to both anger and fear than neutral for the torso and arms. Representational similarity analysis showed that neural patterns of the EBA distinguished different bodily expressions. Furthermore, the torso with arms and whole body had higher similarities in EBA representations relative to the legs and whole body, and to the head and whole body. Taken together, these results indicate that diagnostic body parts (i.e., torso with arms) can communicate bodily expression in a detectable manner.
Collapse
|
48
|
Muukkonen I, Kilpeläinen M, Turkkila R, Saarela T, Salmela V. Obligatory integration of face features in expression discrimination. VISUAL COGNITION 2022. [DOI: 10.1080/13506285.2022.2046222] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Affiliation(s)
- I. Muukkonen
- Department of Psychology and Logopedics, University of Helsinki, Helsinki, Finland
| | - M. Kilpeläinen
- Department of Psychology and Logopedics, University of Helsinki, Helsinki, Finland
| | - R. Turkkila
- Department of Psychology and Logopedics, University of Helsinki, Helsinki, Finland
| | - T. Saarela
- Department of Psychology and Logopedics, University of Helsinki, Helsinki, Finland
| | - V. Salmela
- Department of Psychology and Logopedics, University of Helsinki, Helsinki, Finland
| |
Collapse
|
49
|
Sofer C, Avidan G, Vilenchik D, Dotsch R. The Compositionality of Facial Expressions. Perception 2022; 51:172-186. [PMID: 35230208 DOI: 10.1177/03010066221077573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
The principle of compositionality, an important postulation in language and cognition research, posits that the meaning of a complex expression is determined by the meaning of its constituting parts and the operation performed on those parts. Here, we provide strong evidence that this principle plays a significant role also in interpreting facial expressions. In three studies in which perceivers interpreted sequences of two emotional facial expression images, we show that the composite meaning of facial expressions results from the meaning of its constituting expressions and an algebraic operation performed on those expressions. Our study offers a systematic account as to how the meaning of facial expressions (single and sequences) are being formed and perceived. In a broader context, our results raise the possibility that the principle of compositionality may apply to human communication modalities beyond spoken language, whereby a minimal number of components are expanded to a much greater number of meanings.
Collapse
Affiliation(s)
- Carmel Sofer
- 26732Department of Cognitive & Brain Sciences, Department of Industrial Engineering and Management, Ben Gurion University of the Negev, Israel
| | - Galia Avidan
- 26732Department of Psychology, Ben Gurion University of the Negev, Israel
| | - Dan Vilenchik
- School of Electrical and Computer Engineering, 26732Ben Gurion University of the Negev, Israel
| | | |
Collapse
|
50
|
Leung FYN, Sin J, Dawson C, Ong JH, Zhao C, Veić A, Liu F. Emotion recognition across visual and auditory modalities in autism spectrum disorder: A systematic review and meta-analysis. DEVELOPMENTAL REVIEW 2022. [DOI: 10.1016/j.dr.2021.101000] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
|