1
|
Hackney BC, Pyles JA, Grossman ED. A quantitative comparison of atlas parcellations on the human superior temporal sulcus. Brain Res 2024; 1842:149119. [PMID: 38986829 DOI: 10.1016/j.brainres.2024.149119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2024] [Revised: 06/19/2024] [Accepted: 07/07/2024] [Indexed: 07/12/2024]
Abstract
The superior temporal sulcus (STS) has a functional topography that has been difficult to characterize through traditional approaches. Automated atlas parcellations may be one solution while also being beneficial for both dimensional reduction and standardizing regions of interest, but they yield very different boundary definitions along the STS. Here we evaluate how well machine learning classifiers can correctly identify six social cognitive tasks from STS activation patterns dimensionally reduced using four popular atlases (Glasser et al., 2016; Gordon et al., 2016; Power et al., 2011 as projected onto the surface by Arslan et al., 2018; Schaefer et al., 2018). Functional data was summarized within each STS parcel in one of four ways, then subjected to leave-one-subject-out cross-validation SVM classification. We found that the classifiers could readily label conditions when data was parcellated using any of the four atlases, evidence that dimensional reduction to parcels did not compromise functional fingerprints. Mean activation for the social conditions was the most effective metric for classification in the right STS, whereas all the metrics classified equally well in the left STS. Interestingly, even atlases constructed from random parcellation schemes (null atlases) classified the conditions with high accuracy. We therefore conclude that the complex activation maps on the STS are readily differentiated at a coarse granular level, despite a strict topography having not yet been identified. Further work is required to identify what features have greatest potential to improve the utility of atlases in replacing functional localizers.
Collapse
Affiliation(s)
- Brandon C Hackney
- Department of Cognitive Sciences, University of California, Irvine, 2201 Social & Behavioral Sciences Gateway, Irvine, CA 92697, United States.
| | - John A Pyles
- Department of Psychology, Center for Human Neuroscience, University of Washington, 119 Guthrie Hall, Seattle, WA 98195, United States
| | - Emily D Grossman
- Department of Cognitive Sciences, University of California, Irvine, 2201 Social & Behavioral Sciences Gateway, Irvine, CA 92697, United States
| |
Collapse
|
2
|
Ceccarini F, Colpizzi I, Caudek C. Age-dependent changes in the anger superiority effect: Evidence from a visual search task. Psychon Bull Rev 2024; 31:1704-1713. [PMID: 38238561 PMCID: PMC11358229 DOI: 10.3758/s13423-023-02401-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/03/2023] [Indexed: 04/16/2024]
Abstract
The perception of threatening facial expressions is a critical skill necessary for detecting the emotional states of others and responding appropriately. The anger superiority effect hypothesis suggests that individuals are better at processing and identifying angry faces compared with other nonthreatening facial expressions. In adults, the anger superiority effect is present even after controlling for the bottom-up visual saliency, and when ecologically valid stimuli are used. However, it is as yet unclear whether this effect is present in children. To fill this gap, we tested the anger superiority effect in children ages 6-14 years in a visual search task by using emotional dynamic stimuli and equating the visual salience of target and distractors. The results suggest that in childhood, the angry superiority effect consists of improved accuracy in detecting angry faces, while in adolescence, the ability to discriminate angry faces undergoes further development, enabling faster and more accurate threat detection.
Collapse
Affiliation(s)
| | - Ilaria Colpizzi
- Health Sciences Department, Università Degli Studi Di Firenze, Florence, Italy
| | - Corrado Caudek
- NEUROFARBA Department, Università degli Studi di Firenze, Florence, Italy.
| |
Collapse
|
3
|
Lahner B, Dwivedi K, Iamshchinina P, Graumann M, Lascelles A, Roig G, Gifford AT, Pan B, Jin S, Ratan Murty NA, Kay K, Oliva A, Cichy R. Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nat Commun 2024; 15:6241. [PMID: 39048577 PMCID: PMC11269733 DOI: 10.1038/s41467-024-50310-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 07/04/2024] [Indexed: 07/27/2024] Open
Abstract
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
Collapse
Affiliation(s)
- Benjamin Lahner
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
| | - Kshitij Dwivedi
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
| | - Polina Iamshchinina
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Monika Graumann
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Alex Lascelles
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Gemma Roig
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
- The Hessian Center for AI (hessian.AI), Darmstadt, Germany
| | | | - Bowen Pan
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - SouYoung Jin
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - N Apurva Ratan Murty
- Department of Brain and Cognitive Science, MIT, Cambridge, MA, USA
- School of Psychology, Georgia Institute of Technology, Atlanta, GA, USA
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Radoslaw Cichy
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| |
Collapse
|
4
|
Mudrik L, Hirschhorn R, Korisky U. Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes. Neuron 2024; 112:1642-1656. [PMID: 38653247 PMCID: PMC11100345 DOI: 10.1016/j.neuron.2024.03.031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 03/23/2024] [Accepted: 03/29/2024] [Indexed: 04/25/2024]
Abstract
The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.
Collapse
Affiliation(s)
- Liad Mudrik
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel; Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
| | - Rony Hirschhorn
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| | - Uri Korisky
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
5
|
Tully LM, Blendermann M, Fine JR, Zakskorn LN, Fritz M, Hamlett GE, Lamb ST, Moody AK, Ng J, Parakul N, Ritter BM, Rahim R, Yu G, Taylor SL. The SocialVidStim: a video database of positive and negative social evaluation stimuli for use in social cognitive neuroscience paradigms. Soc Cogn Affect Neurosci 2024; 19:nsae024. [PMID: 38597895 PMCID: PMC11015894 DOI: 10.1093/scan/nsae024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2021] [Revised: 01/06/2023] [Accepted: 04/05/2024] [Indexed: 04/11/2024] Open
Abstract
This paper describes the SocialVidStim-a database of video stimuli available to the scientific community depicting positive and negative social evaluative and neutral statements. The SocialVidStim comprises 53 diverse individuals reflecting the demographic makeup of the USA, ranging from 9 to 41 years old, saying 20-60 positive and 20-60 negative social evaluative statements (e.g. 'You are a very trustworthy/annoying person'), and 20-60 neutral statements (e.g. 'The sky is blue'), totaling 5793 videos post-production. The SocialVidStim are designed for use in behavioral and functional magetic resonance imaging paradigms, across developmental stages, in diverse populations. This study describes stimuli development and reports initial validity and reliability data on a subset videos (N = 1890) depicting individuals aged 18-41 years. Raters perceive videos as expected: positive videos elicit positively valenced ratings, negative videos elicit negatively valenced ratings and neutral videos are rated as neutral. Test-retest reliability data demonstrate intraclass correlations in the good-to-excellent range for negative and positive videos and the moderate range for neutral videos. We also report small effects on valence and arousal that should be considered during stimuli selection, including match between rater and actor sex and actor believability. The SocialVidStim is a resource for researchers and we offer suggestions for using the SocialVidStim in future research.
Collapse
Affiliation(s)
- Laura M Tully
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Mary Blendermann
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Jeffrey R Fine
- Division of Biostatistics, Department of Public Health Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Lauren N Zakskorn
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Matilda Fritz
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Gabriella E Hamlett
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Shannon T Lamb
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Anna K Moody
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Julenne Ng
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Narimes Parakul
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Bryn M Ritter
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Raisa Rahim
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Grace Yu
- Department of Psychiatry and Behavioral Sciences, University of California, Davis, Sacramento, CA 95817, USA
| | - Sandra L Taylor
- Division of Biostatistics, Department of Public Health Sciences, University of California, Davis, Sacramento, CA 95817, USA
| |
Collapse
|
6
|
Richoz AR, Stacchi L, Schaller P, Lao J, Papinutto M, Ticcinelli V, Caldara R. Recognizing facial expressions of emotion amid noise: A dynamic advantage. J Vis 2024; 24:7. [PMID: 38197738 PMCID: PMC10790674 DOI: 10.1167/jov.24.1.7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 11/12/2023] [Indexed: 01/11/2024] Open
Abstract
Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.
Collapse
Affiliation(s)
- Anne-Raphaëlle Richoz
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Lisa Stacchi
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Pauline Schaller
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Junpeng Lao
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Michael Papinutto
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Valentina Ticcinelli
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| |
Collapse
|
7
|
Jangraw DC, Finn ES, Bandettini PA, Landi N, Sun H, Hoeft F, Chen G, Pugh KR, Molfese PJ. Inter-subject correlation during long narratives reveals widespread neural correlates of reading ability. Neuroimage 2023; 282:120390. [PMID: 37751811 PMCID: PMC10783814 DOI: 10.1016/j.neuroimage.2023.120390] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Revised: 09/19/2023] [Accepted: 09/23/2023] [Indexed: 09/28/2023] Open
Abstract
Recent work using fMRI inter-subject correlation analysis has provided new information about the brain's response to video and audio narratives, particularly in frontal regions not typically activated by single words. This approach is very well suited to the study of reading, where narrative is central to natural experience. But since past reading paradigms have primarily presented single words or phrases, the influence of narrative on semantic processing in the brain - and how that influence might change with reading ability - remains largely unexplored. In this study, we presented coherent stories to adolescents and young adults with a wide range of reading abilities. The stories were presented in alternating visual and auditory blocks. We used a dimensional inter-subject correlation analysis to identify regions in which better and worse readers had varying levels of consistency with other readers. This analysis identified a widespread set of brain regions in which activity timecourses were more similar among better readers than among worse readers. These differences were not detected with standard block activation analyses. Worse readers had higher correlation with better readers than with other worse readers, suggesting that the worse readers had "idiosyncratic" responses rather than using a single compensatory mechanism. Close inspection confirmed that these differences were not explained by differences in IQ or motion. These results suggest an expansion of the current view of where and how reading ability is reflected in the brain, and in doing so, they establish inter-subject correlation as a sensitive tool for future studies of reading disorders.
Collapse
Affiliation(s)
- David C Jangraw
- Section on Functional Imaging Methods, NIMH, Bethesda, MD, United States; Emotion and Development Branch, NIMH, Bethesda, MD, United States; Department of Electrical and Biomedical Engineering, University of Vermont, Burlington, VT, United States.
| | - Emily S Finn
- Section on Functional Imaging Methods, NIMH, Bethesda, MD, United States; Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, United States
| | - Peter A Bandettini
- Section on Functional Imaging Methods, NIMH, Bethesda, MD, United States; Center for Multimodal Neuroimaging, NIMH, Bethesda, MD, United States
| | - Nicole Landi
- Haskins Laboratories, New Haven, CT, United States
| | - Haorui Sun
- Department of Electrical and Biomedical Engineering, University of Vermont, Burlington, VT, United States
| | - Fumiko Hoeft
- Haskins Laboratories, New Haven, CT, United States; Department of Psychological Sciences, University of Connecticut, Hartford, CT, United States
| | - Gang Chen
- Statistical Computing Core, NIMH, Bethesda, MD, United States
| | - Kenneth R Pugh
- Haskins Laboratories, New Haven, CT, United States; Department of Pediatrics, Yale University School of Medicine, New Haven, CT, United States
| | - Peter J Molfese
- Center for Multimodal Neuroimaging, NIMH, Bethesda, MD, United States; Haskins Laboratories, New Haven, CT, United States
| |
Collapse
|
8
|
Pitcher D, Ianni GR, Holiday K, Ungerleider LG. Identifying the cortical face network with dynamic face stimuli: A large group fMRI study. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.26.559583. [PMID: 37886588 PMCID: PMC10602036 DOI: 10.1101/2023.09.26.559583] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/28/2023]
Abstract
Functional magnetic resonance imaging (fMRI) studies have identified a network of face-selective regions distributed across the human brain. In the present study, we analyzed data from a large group of gender-balanced participants to investigate how reliably these face-selective regions could be identified across both cerebral hemispheres. Participants ( N =52) were scanned with fMRI while viewing short videos of faces, bodies, and objects. Results revealed that five face-selective regions: the fusiform face area (FFA), posterior superior temporal sulcus (pSTS), anterior superior temporal sulcus (aSTS), inferior frontal gyrus (IFG) and the amygdala were all larger in the right than in the left hemisphere. The occipital face area (OFA) was larger in the right hemisphere as well, but the difference between the hemispheres was not significant. The neural response to moving faces was also greater in face-selective regions in the right than in the left hemisphere. An additional analysis revealed that the pSTS and IFG were significantly larger in the right hemisphere compared to other face-selective regions. This pattern of results demonstrates that moving faces are preferentially processed in the right hemisphere and that the pSTS and IFG appear to be the strongest drivers of this laterality. An analysis of gender revealed that face-selective regions were typically larger in females ( N =26) than males ( N =26), but this gender difference was not statistically significant.
Collapse
|
9
|
Yeung SC, Sidhu J, Youn S, Schaefer HRH, Barton JJS, Corrow SL. The role of the upper and lower face in the recognition of facial identity in dynamic stimuli. Vision Res 2023; 206:108194. [PMID: 36801665 PMCID: PMC10085847 DOI: 10.1016/j.visres.2023.108194] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2021] [Revised: 02/03/2023] [Accepted: 02/04/2023] [Indexed: 02/18/2023]
Abstract
Studies with static faces find that upper face halves are more easily recognized than lower face halves-an upper-face advantage. However, faces are usually encountered as dynamic stimuli, and there is evidence that dynamic information influences face identity recognition. This raises the question of whether dynamic faces also show an upper-face advantage. The objective of this study was to examine whether familiarity for recently learned faces was more accurate for upper or lower face halves, and whether this depended upon whether the face was presented as static or dynamic. In Experiment 1, subjects learned a total of 12 faces--6 static images and 6 dynamic video-clips of actors in silent conversation. In experiment 2, subjects learned 12 faces, all dynamic video-clips. During the testing phase of Experiments 1 (between subjects) and 2 (within subjects), subjects were asked to recognize upper and lower face halves from either static images and/or dynamic clips. The data did not provide evidence for a difference in the upper-face advantage between static and dynamic faces. However, in both experiments, we found an upper-face advantage, consistent with prior literature, for female faces, but not for male faces. In conclusion, the use of dynamic stimuli may have little effect on the presence of an upper-face advantage, especially when the static comparison contains a series of static images, rather than a single static image, and is of sufficient image quality. Future studies could investigate the influence of face gender on the presence of an upper-face advantage.
Collapse
Affiliation(s)
- Shanna C Yeung
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada
| | - Jhunam Sidhu
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada
| | - Sena Youn
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada
| | - Heidi R H Schaefer
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada
| | - Jason J S Barton
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada
| | - Sherryse L Corrow
- Psychology Department, Bethel University, 3900 Bethel Drive, St Paul, MN 55112, Canada.
| |
Collapse
|
10
|
Jeganathan J, Campbell M, Hyett M, Parker G, Breakspear M. Quantifying dynamic facial expressions under naturalistic conditions. eLife 2022; 11:e79581. [PMID: 36043464 PMCID: PMC9439684 DOI: 10.7554/elife.79581] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 08/24/2022] [Indexed: 11/15/2022] Open
Abstract
Facial affect is expressed dynamically - a giggle, grimace, or an agitated frown. However, the characterisation of human affect has relied almost exclusively on static images. This approach cannot capture the nuances of human communication or support the naturalistic assessment of affective disorders. Using the latest in machine vision and systems modelling, we studied dynamic facial expressions of people viewing emotionally salient film clips. We found that the apparent complexity of dynamic facial expressions can be captured by a small number of simple spatiotemporal states - composites of distinct facial actions, each expressed with a unique spectral fingerprint. Sequential expression of these states is common across individuals viewing the same film stimuli but varies in those with the melancholic subtype of major depressive disorder. This approach provides a platform for translational research, capturing dynamic facial expressions under naturalistic conditions and enabling new quantitative tools for the study of affective disorders and related mental illnesses.
Collapse
Affiliation(s)
- Jayson Jeganathan
- School of Psychology, College of Engineering, Science and the Environment, University of NewcastleNewcastleAustralia
- Hunter Medical Research InstituteNewcastleAustralia
| | - Megan Campbell
- School of Psychology, College of Engineering, Science and the Environment, University of NewcastleNewcastleAustralia
- Hunter Medical Research InstituteNewcastleAustralia
| | - Matthew Hyett
- School of Psychological Sciences, University of Western AustraliaPerthAustralia
| | - Gordon Parker
- School of Psychiatry, University of New South WalesKensingtonAustralia
| | - Michael Breakspear
- School of Psychology, College of Engineering, Science and the Environment, University of NewcastleNewcastleAustralia
- Hunter Medical Research InstituteNewcastleAustralia
- School of Medicine and Public Health, College of Medicine, Health and Wellbeing, University of NewcastleNewcastleAustralia
| |
Collapse
|
11
|
Nikel L, Sliwinska MW, Kucuk E, Ungerleider LG, Pitcher D. Measuring the response to visually presented faces in the human lateral prefrontal cortex. Cereb Cortex Commun 2022; 3:tgac036. [PMID: 36159205 PMCID: PMC9491845 DOI: 10.1093/texcom/tgac036] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2022] [Revised: 08/12/2022] [Accepted: 08/14/2022] [Indexed: 12/04/2022] Open
Abstract
Neuroimaging studies identify multiple face-selective areas in the human brain. In the current study, we compared the functional response of the face area in the lateral prefrontal cortex to that of other face-selective areas. In Experiment 1, participants (n = 32) were scanned viewing videos containing faces, bodies, scenes, objects, and scrambled objects. We identified a face-selective area in the right inferior frontal gyrus (rIFG). In Experiment 2, participants (n = 24) viewed the same videos or static images. Results showed that the rIFG, right posterior superior temporal sulcus (rpSTS), and right occipital face area (rOFA) exhibited a greater response to moving than static faces. In Experiment 3, participants (n = 18) viewed face videos in the contralateral and ipsilateral visual fields. Results showed that the rIFG and rpSTS showed no visual field bias, while the rOFA and right fusiform face area (rFFA) showed a contralateral bias. These experiments suggest two conclusions; firstly, in all three experiments, the face area in the IFG was not as reliably identified as face areas in the occipitotemporal cortex. Secondly, the similarity of the response profiles in the IFG and pSTS suggests the areas may perform similar cognitive functions, a conclusion consistent with prior neuroanatomical and functional connectivity evidence.
Collapse
Affiliation(s)
- Lara Nikel
- Department of Psychology, University of York, Heslington , York YO10 5DD , UK
| | | | - Emel Kucuk
- Department of Psychology, University of York, Heslington , York YO10 5DD , UK
| | - Leslie G Ungerleider
- Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health , Bethesda, MD, 20892 , USA
| | - David Pitcher
- Department of Psychology, University of York, Heslington , York YO10 5DD , UK
| |
Collapse
|
12
|
Sun Y, Ma J, Huang M, Yi Y, Wang Y, Gu Y, Lin Y, Li LMW, Dai Z. Functional connectivity dynamics as a function of the fluctuation of tension during film watching. Brain Imaging Behav 2022; 16:1260-1274. [PMID: 34988779 DOI: 10.1007/s11682-021-00593-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/12/2021] [Indexed: 11/28/2022]
Abstract
To advance the understanding of the dynamic relationship between brain activities and emotional experiences, we examined the neural patterns of tension, a unique emotion that highly depends on how an event unfolds. Specifically, the present study explored the temporal relationship between functional connectivity patterns within and between different brain functional modules and the fluctuation in tension during film watching. Due to the highly contextualized and time-varying nature of tension, we expected that multiple neural networks would be involved in the dynamic tension experience. Using the neuroimaging data of 546 participants, we conducted a dynamic brain analysis to identify the intra- and inter-module functional connectivity patterns that are significantly correlated with the fluctuation of tension over time. The results showed that the inter-module connectivity of cingulo-opercular network, fronto-parietal network, and default mode network is involved in the dynamic experience of tension. These findings demonstrate a close relationship between brain functional connectivity patterns and emotional dynamics, which supports the importance of functional connectivity dynamics in understanding our cognitive and emotional processes.
Collapse
Affiliation(s)
- Yadi Sun
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Junji Ma
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Miner Huang
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Yangyang Yi
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Yiheng Wang
- Institute of Applied Psychology, Guangdong University of Finance, Guangzhou, 510006, China
| | - Yue Gu
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Ying Lin
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China
| | - Liman Man Wai Li
- Department of Psychology and Centre for Psychosocial Health, The Education University of Hong Kong, Hong Kong SAR, China.
| | - Zhengjia Dai
- Department of Psychology, Sun Yat-sen University, Guangzhou, 510006, China.
| |
Collapse
|
13
|
Richey JA, Gracanin D, LaConte S, Lisinski J, Kim I, Coffman M, Antezana L, Carlton CN, Garcia KM, White SW. Neural Mechanisms of Facial Emotion Recognition in Autism: Distinct Roles for Anterior Cingulate and dlPFC. JOURNAL OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY : THE OFFICIAL JOURNAL FOR THE SOCIETY OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY, AMERICAN PSYCHOLOGICAL ASSOCIATION, DIVISION 53 2022; 51:323-343. [PMID: 35476602 PMCID: PMC9177800 DOI: 10.1080/15374416.2022.2051528] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
OBJECTIVE The present study sought to measure and internally validate neural markers of facial emotion recognition (FER) in adolescents and young adults with ASD to inform targeted intervention. METHOD We utilized fMRI to measure patterns of brain activity among individuals with ASD (N = 21) and matched controls (CON; N = 20) 2 s prior to judgments about the identity of six distinct facial emotions (happy, sad, angry, surprised, fearful, disgust). RESULTS Predictive modeling of fMRI data (support vector classification; SVC) identified mechanistic roles for brain regions that forecasted correct and incorrect identification of facial emotion as well as sources of errors over these decisions. BOLD signal activation in bilateral insula, anterior cingulate (ACC) and right dorsolateral prefrontal cortex (dlPFC) preceded accurate FER in both controls and ASD. Predictive modeling utilizing SVC confirmed the utility of ACC in forecasting correct decisions in controls but not ASD, and further indicated that a region within the right dlPFC was the source of a type 1 error signal in ASD (i.e. neural marker reflecting an impending correct judgment followed by an incorrect behavioral response) approximately two seconds prior to emotion judgments during fMRI. CONCLUSIONS ACC forecasted correct decisions only among control participants. Right dlPFC was the source of a false-positive signal immediately prior to an error about the nature of a facial emotion in adolescents and young adults with ASD, potentially consistent with prior work indicating that dlPFC may play a role in attention to and regulation of emotional experience.
Collapse
Affiliation(s)
- John A. Richey
- Department of Psychology, Virginia Tech. 109 Williams Hall, MC0436, Blacksburg, VA 24061
| | - Denis Gracanin
- Department of Computer Science, Virginia Tech. 2202 Kraft Drive, Room 1135, Blacksburg VA 24060
| | - Stephen LaConte
- Fralin Biomedical Research Institute at Virginia Tech Carilion. 2 Riverside Circle Roanoke, VA 24016
- Department of Biomedical Engineering and Mechanics, Virginia Tech
| | - Jonathan Lisinski
- Fralin Biomedical Research Institute at Virginia Tech Carilion. 2 Riverside Circle Roanoke, VA 24016
| | - Inyoung Kim
- Fralin Biomedical Research Institute at Virginia Tech Carilion. 2 Riverside Circle Roanoke, VA 24016
- Department of Statistics, Hutcheson Hall, RM 406-A Virginia Tech. Blacksburg, VA 24061
| | - Marika Coffman
- Department of Psychology, Virginia Tech. 109 Williams Hall, MC0436, Blacksburg, VA 24061
- Duke University Center for Autism and Brain Development. 2608 Erwin Rd, Suite 300 b
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, 27701Durham, NC 27705
| | - Ligia Antezana
- Department of Psychology, Virginia Tech. 109 Williams Hall, MC0436, Blacksburg, VA 24061
| | - Corinne N. Carlton
- Department of Psychology, Virginia Tech. 109 Williams Hall, MC0436, Blacksburg, VA 24061
| | - Katelyn M. Garcia
- Department of Psychology, Virginia Tech. 109 Williams Hall, MC0436, Blacksburg, VA 24061
| | - Susan W. White
- Center for Youth Development and Intervention, McMillan Building 101-F, University of Alabama. Tuscaloosa, AL 35487
| |
Collapse
|
14
|
Furl N, Begum F, Ferrarese FP, Jans S, Woolley C, Sulik J. Caricatured facial movements enhance perception of emotional facial expressions. Perception 2022; 51:313-343. [PMID: 35341407 PMCID: PMC9017061 DOI: 10.1177/03010066221086452] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Although faces “in the wild” constantly undergo complicated movements, humans adeptly
perceive facial identity and expression. Previous studies, focusing mainly on identity,
used photographic caricature to show that distinctive form increases perceived
dissimilarity. We tested whether distinctive facial movements showed
similar effects, and we focussed on both perception of expression and
identity. We caricatured the movements of an animated computer head,
using physical motion metrics extracted from videos. We verified that these “ground truth”
metrics showed the expected effects: Caricature increased physical dissimilarity between
faces differing in expression and those differing in identity. Like the ground truth
dissimilarity, participants’ dissimilarity perception was increased by caricature when
faces differed in expression. We found these perceived dissimilarities to reflect the
“representational geometry” of the ground truth. However, neither of these findings held
for faces differing in identity. These findings replicated across two paradigms: pairwise
ratings and multiarrangement. In a final study, motion caricature did not improve
recognition memory for identity, whether manipulated at study or test. We report several
forms of converging evidence for spatiotemporal caricature effects on dissimilarity
perception of different expressions. However, more work needs to be done to discover what
identity-specific movements can enhance face identification.
Collapse
Affiliation(s)
| | | | | | - Sarah Jans
- Royal Holloway, 3162University of London, UK
| | | | - Justin Sulik
- Royal Holloway, 3162University of London, UK; Cognition, Values & Behavior, Ludwig Maximilian University of Munich, Germany
| |
Collapse
|
15
|
Alp N, Ozkan H. Neural correlates of integration processes during dynamic face perception. Sci Rep 2022; 12:118. [PMID: 34996892 PMCID: PMC8742062 DOI: 10.1038/s41598-021-02808-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2021] [Accepted: 11/22/2021] [Indexed: 11/10/2022] Open
Abstract
Integrating the spatiotemporal information acquired from the highly dynamic world around us is essential to navigate, reason, and decide properly. Although this is particularly important in a face-to-face conversation, very little research to date has specifically examined the neural correlates of temporal integration in dynamic face perception. Here we present statistically robust observations regarding the brain activations measured via electroencephalography (EEG) that are specific to the temporal integration. To that end, we generate videos of neutral faces of individuals and non-face objects, modulate the contrast of the even and odd frames at two specific frequencies (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$f_1$$\end{document}f1 and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$f_2$$\end{document}f2) in an interlaced manner, and measure the steady-state visual evoked potential as participants view the videos. Then, we analyze the intermodulation components (IMs: (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$nf_1\pm mf_2$$\end{document}nf1±mf2), a linear combination of the fundamentals with integer multipliers) that consequently reflect the nonlinear processing and indicate temporal integration by design. We show that electrodes around the medial temporal, inferior, and medial frontal areas respond strongly and selectively when viewing dynamic faces, which manifests the essential processes underlying our ability to perceive and understand our social world. The generation of IMs is only possible if even and odd frames are processed in succession and integrated temporally, therefore, the strong IMs in our frequency spectrum analysis show that the time between frames (1/60 s) is sufficient for temporal integration.
Collapse
Affiliation(s)
- Nihan Alp
- Psychology, Sabanci University, Istanbul, Turkey.
| | - Huseyin Ozkan
- Electronics Engineering, Sabanci University, Istanbul, Turkey
| |
Collapse
|
16
|
Kobayashi M, Kanazawa S, Yamaguchi MK, O'Toole AJ. Cortical processing of dynamic bodies in the superior occipito-temporal regions of the infants' brain: Difference from dynamic faces and inversion effect. Neuroimage 2021; 244:118598. [PMID: 34587515 DOI: 10.1016/j.neuroimage.2021.118598] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2021] [Revised: 09/15/2021] [Accepted: 09/17/2021] [Indexed: 11/17/2022] Open
Abstract
Previous functional neuroimaging studies imply a crucial role of the superior temporal regions (e.g., superior temporal sulcus: STS) for processing of dynamic faces and bodies. However, little is known about the cortical processing of moving faces and bodies in infancy. The current study used functional near-infrared spectroscopy (fNIRS) to directly compare cortical hemodynamic responses to dynamic faces (videos of approaching people with blurred bodies) and dynamic bodies (videos of approaching people with blurred faces) in infants' brain. We also examined the body-inversion effect in 5- to 8-month-old infants using hemodynamic responses as a measure. We found significant brain activity for the dynamic faces and bodies in the superior area of bilateral temporal cortices in both 5- to 6-month-old and 7- to 8-month-old infants. The hemodynamic responses to dynamic faces occurred across a broader area of cortex in 7- to 8-month-olds than in 5- to 6-month-olds, but we did not find a developmental change for dynamic bodies. There was no significant activation when the stimuli were presented upside down, indicating that these activation patterns did not result from the low-level visual properties of dynamic faces and bodies. Additionally, we found that the superior temporal regions showed a body inversion effect in infants aged over 5 months: the upright dynamic body stimuli induced stronger activation compared to the inverted stimuli. The most important contribution of the present study is that we identified cortical areas responsive to dynamic bodies and faces in two groups of infants (5-6-months and 7-8-months of age) and we found different developmental trends for the processing of bodies and faces.
Collapse
Affiliation(s)
- Megumi Kobayashi
- Department of Functioning and Disability, Institute for Developmental Research, Aichi Developmental Disability Center, Japan.
| | - So Kanazawa
- Department of Psychology, Japan Women's University, Japan
| | | | - Alice J O'Toole
- School of Behavioral and Brain Sciences, The University of Texas at Dallas, USA
| |
Collapse
|
17
|
Liu M, Liu CH, Zheng S, Zhao K, Fu X. Reexamining the neural network involved in perception of facial expression: A meta-analysis. Neurosci Biobehav Rev 2021; 131:179-191. [PMID: 34536463 DOI: 10.1016/j.neubiorev.2021.09.024] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2021] [Revised: 08/19/2021] [Accepted: 09/09/2021] [Indexed: 11/15/2022]
Abstract
Perception of facial expression is essential for social interactions. Although a few competing models have enjoyed some success to map brain regions, they are also facing difficult challenges. The current study used an updated activation likelihood estimation (ALE) method of meta-analysis to explore the involvement of brain regions in facial expression processing. The sample contained 96 functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) studies of healthy adults with the results of whole-brain analyses. The key findings revealed that the ventral pathway, especially the left fusiform face area (FFA) region, was more responsive to facial expression. The left posterior FFA showed strong involvement when participants passively viewing emotional faces without being asked to judge the type of expression or other attributes of the stimuli. Through meta-analytic connectivity modeling (MACM) of the main brain regions in the ventral pathway, we constructed a co-activating neural network as a revised model of facial expression processing that assigns prominent roles to the amygdala, FFA, the occipital gyrus, and the inferior frontal gyrus.
Collapse
Affiliation(s)
- Mingtong Liu
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Chang Hong Liu
- Department of Psychology, Bournemouth University, Dorset, United Kingdom
| | - Shuang Zheng
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Ke Zhao
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Xiaolan Fu
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China.
| |
Collapse
|
18
|
Pitcher D, Pilkington A, Rauth L, Baker C, Kravitz DJ, Ungerleider LG. The Human Posterior Superior Temporal Sulcus Samples Visual Space Differently From Other Face-Selective Regions. Cereb Cortex 2021; 30:778-785. [PMID: 31264693 DOI: 10.1093/cercor/bhz125] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2019] [Revised: 05/17/2019] [Accepted: 05/20/2019] [Indexed: 01/22/2023] Open
Abstract
Neuroimaging studies show that ventral face-selective regions, including the fusiform face area (FFA) and occipital face area (OFA), preferentially respond to faces presented in the contralateral visual field (VF). In the current study we measured the VF response of the face-selective posterior superior temporal sulcus (pSTS). Across 3 functional magnetic resonance imaging experiments, participants viewed face videos presented in different parts of the VF. Consistent with prior results, we observed a contralateral VF bias in bilateral FFA, right OFA (rOFA), and bilateral human motion-selective area MT+. Intriguingly, this contralateral VF bias was absent in the bilateral pSTS. We then delivered transcranial magnetic stimulation (TMS) over right pSTS (rpSTS) and rOFA, while participants matched facial expressions in both hemifields. TMS delivered over the rpSTS disrupted performance in both hemifields, but TMS delivered over the rOFA disrupted performance in the contralateral hemifield only. These converging results demonstrate that the contralateral bias for faces observed in ventral face-selective areas is absent in the pSTS. This difference in VF response is consistent with face processing models proposing 2 functionally distinct pathways. It further suggests that these models should account for differences in interhemispheric connections between the face-selective areas across these 2 pathways.
Collapse
Affiliation(s)
- David Pitcher
- Department of Psychology, University of York, Heslington, York YO105DD, UK.,Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD 20892, USA
| | - Amy Pilkington
- Department of Psychology, University of York, Heslington, York YO105DD, UK
| | - Lionel Rauth
- Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD 20892, USA
| | - Chris Baker
- Section on Learning and Plasticity, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD 20892, USA
| | - Dwight J Kravitz
- Department of Psychology, George Washington University, 2125 G Street NW, Washington, DC 20052, USA
| | - Leslie G Ungerleider
- Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD 20892, USA
| |
Collapse
|
19
|
Maguinness C, von Kriegstein K. Visual mechanisms for voice-identity recognition flexibly adjust to auditory noise level. Hum Brain Mapp 2021; 42:3963-3982. [PMID: 34043249 PMCID: PMC8288083 DOI: 10.1002/hbm.25532] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2021] [Revised: 04/26/2021] [Accepted: 05/02/2021] [Indexed: 11/24/2022] Open
Abstract
Recognising the identity of voices is a key ingredient of communication. Visual mechanisms support this ability: recognition is better for voices previously learned with their corresponding face (compared to a control condition). This so‐called ‘face‐benefit’ is supported by the fusiform face area (FFA), a region sensitive to facial form and identity. Behavioural findings indicate that the face‐benefit increases in noisy listening conditions. The neural mechanisms for this increase are unknown. Here, using functional magnetic resonance imaging, we examined responses in face‐sensitive regions while participants recognised the identity of auditory‐only speakers (previously learned by face) in high (SNR −4 dB) and low (SNR +4 dB) levels of auditory noise. We observed a face‐benefit in both noise levels, for most participants (16 of 21). In high‐noise, the recognition of face‐learned speakers engaged the right posterior superior temporal sulcus motion‐sensitive face area (pSTS‐mFA), a region implicated in the processing of dynamic facial cues. The face‐benefit in high‐noise also correlated positively with increased functional connectivity between this region and voice‐sensitive regions in the temporal lobe in the group of 16 participants with a behavioural face‐benefit. In low‐noise, the face‐benefit was robustly associated with increased responses in the FFA and to a lesser extent the right pSTS‐mFA. The findings highlight the remarkably adaptive nature of the visual network supporting voice‐identity recognition in auditory‐only listening conditions.
Collapse
Affiliation(s)
- Corrina Maguinness
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.,Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Katharina von Kriegstein
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.,Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| |
Collapse
|
20
|
Khosla M, Ngo GH, Jamison K, Kuceyeski A, Sabuncu MR. Cortical response to naturalistic stimuli is largely predictable with deep neural networks. SCIENCE ADVANCES 2021; 7:7/22/eabe7547. [PMID: 34049888 PMCID: PMC8163078 DOI: 10.1126/sciadv.abe7547] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/11/2020] [Accepted: 04/12/2021] [Indexed: 05/08/2023]
Abstract
Naturalistic stimuli, such as movies, activate a substantial portion of the human brain, invoking a response shared across individuals. Encoding models that predict neural responses to arbitrary stimuli can be very useful for studying brain function. However, existing models focus on limited aspects of naturalistic stimuli, ignoring the dynamic interactions of modalities in this inherently context-rich paradigm. Using movie-watching data from the Human Connectome Project, we build group-level models of neural activity that incorporate several inductive biases about neural information processing, including hierarchical processing, temporal assimilation, and auditory-visual interactions. We demonstrate how incorporating these biases leads to remarkable prediction performance across large areas of the cortex, beyond the sensory-specific cortices into multisensory sites and frontal cortex. Furthermore, we illustrate that encoding models learn high-level concepts that generalize to task-bound paradigms. Together, our findings underscore the potential of encoding models as powerful tools for studying brain function in ecologically valid conditions.
Collapse
Affiliation(s)
- Meenakshi Khosla
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA
| | - Gia H Ngo
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA
| | - Keith Jamison
- Department of Radiology, Weill Cornell Medicine, New York, NY, USA
| | - Amy Kuceyeski
- Department of Radiology, Weill Cornell Medicine, New York, NY, USA
- Brain and Mind Research Institute, Weill Cornell Medicine, New York, NY, USA
| | - Mert R Sabuncu
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA.
- Department of Radiology, Weill Cornell Medicine, New York, NY, USA
- Nancy E. & Peter C. Meinig School of Biomedical Engineering, Cornell University, Ithaca, NY, USA
| |
Collapse
|
21
|
Pitcher D, Ungerleider LG. Evidence for a Third Visual Pathway Specialized for Social Perception. Trends Cogn Sci 2021; 25:100-110. [PMID: 33334693 PMCID: PMC7811363 DOI: 10.1016/j.tics.2020.11.006] [Citation(s) in RCA: 173] [Impact Index Per Article: 57.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 11/18/2020] [Accepted: 11/18/2020] [Indexed: 11/20/2022]
Abstract
Existing models propose that primate visual cortex is divided into two functionally distinct pathways. The ventral pathway computes the identity of an object; the dorsal pathway computes the location of an object, and the actions related to that object. Despite remaining influential, the two visual pathways model requires revision. Both human and non-human primate studies reveal the existence of a third visual pathway on the lateral brain surface. This third pathway projects from early visual cortex, via motion-selective areas, into the superior temporal sulcus (STS). Studies demonstrating that the STS computes the actions of moving faces and bodies (e.g., expressions, eye-gaze, audio-visual integration, intention, and mood) show that the third visual pathway is specialized for the dynamic aspects of social perception.
Collapse
Affiliation(s)
- David Pitcher
- Department of Psychology, University of York, York, YO10 5DD, UK.
| | - Leslie G Ungerleider
- Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD 20892, USA
| |
Collapse
|
22
|
Hendriks MHA, Dillen C, Vettori S, Vercammen L, Daniels N, Steyaert J, Op de Beeck H, Boets B. Neural processing of facial identity and expression in adults with and without autism: A multi-method approach. NEUROIMAGE-CLINICAL 2020; 29:102520. [PMID: 33338966 PMCID: PMC7750419 DOI: 10.1016/j.nicl.2020.102520] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Revised: 10/23/2020] [Accepted: 11/30/2020] [Indexed: 11/28/2022]
Abstract
The ability to recognize faces and facial expressions is a common human talent. It has, however, been suggested to be impaired in individuals with autism spectrum disorder (ASD). The goal of this study was to compare the processing of facial identity and emotion between individuals with ASD and neurotypicals (NTs). Behavioural and functional magnetic resonance imaging (fMRI) data from 46 young adults (aged 17-23 years, NASD = 22, NNT = 24) was analysed. During fMRI data acquisition, participants discriminated between short clips of a face transitioning from a neutral to an emotional expression. Stimuli included four identities and six emotions. We performed behavioural, univariate, multi-voxel, adaptation and functional connectivity analyses to investigate potential group differences. The ASD-group did not differ from the NT-group on behavioural identity and expression processing tasks. At the neural level, we found no differences in average neural activation, neural activation patterns and neural adaptation to faces in face-related brain regions. In terms of functional connectivity, we found that amygdala seems to be more strongly connected to inferior occipital cortex and V1 in individuals with ASD. Overall, the findings indicate that neural representations of facial identity and expression have a similar quality in individuals with and without ASD, but some regions containing these representations are connected differently in the extended face processing network.
Collapse
Affiliation(s)
- Michelle H A Hendriks
- Department of Brain and Cognition, KU Leuven, Tiensestraat 102 - bus 3714, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium
| | - Claudia Dillen
- Department of Brain and Cognition, KU Leuven, Tiensestraat 102 - bus 3714, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium
| | - Sofie Vettori
- Centre for Developmental Psychiatry, KU Leuven, Kapucijnenvoer 7 blok h - bus 7001, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium
| | - Laura Vercammen
- Department of Brain and Cognition, KU Leuven, Tiensestraat 102 - bus 3714, Leuven, Belgium
| | - Nicky Daniels
- Department of Brain and Cognition, KU Leuven, Tiensestraat 102 - bus 3714, Leuven, Belgium; Centre for Developmental Psychiatry, KU Leuven, Kapucijnenvoer 7 blok h - bus 7001, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium
| | - Jean Steyaert
- Centre for Developmental Psychiatry, KU Leuven, Kapucijnenvoer 7 blok h - bus 7001, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium
| | - Hans Op de Beeck
- Department of Brain and Cognition, KU Leuven, Tiensestraat 102 - bus 3714, Leuven, Belgium; Leuven Brain Institute, KU Leuven, Leuven, Belgium
| | - Bart Boets
- Centre for Developmental Psychiatry, KU Leuven, Kapucijnenvoer 7 blok h - bus 7001, Leuven, Belgium; Leuven Brain Institute, KU Leuven, Leuven, Belgium; Leuven Autism Research Consortium, KU Leuven, Leuven, Belgium.
| |
Collapse
|
23
|
Hildesheim FE, Debus I, Kessler R, Thome I, Zimmermann KM, Steinsträter O, Sommer J, Kamp-Becker I, Stark R, Jansen A. The Trajectory of Hemispheric Lateralization in the Core System of Face Processing: A Cross-Sectional Functional Magnetic Resonance Imaging Pilot Study. Front Psychol 2020; 11:507199. [PMID: 33123034 PMCID: PMC7566903 DOI: 10.3389/fpsyg.2020.507199] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2020] [Accepted: 09/10/2020] [Indexed: 12/12/2022] Open
Abstract
Face processing is mediated by a distributed neural network commonly divided into a “core system” and an “extended system.” The core system consists of several, typically right-lateralized brain regions in the occipito-temporal cortex, including the occipital face area (OFA), the fusiform face area (FFA) and the posterior superior temporal sulcus (pSTS). It was recently proposed that the face processing network is initially bilateral and becomes right-specialized in the course of the development of reading abilities due to the competition between language-related regions in the left occipito-temporal cortex (e.g., the visual word form area, VWFA) and the FFA for common neural resources. In the present pilot study, we assessed the neural face processing network in 12 children (aged 7–9 years) and 10 adults with functional magnetic resonance imaging (fMRI). The hemispheric lateralization of the core face regions was compared between both groups. The study had two goals: First, we aimed to establish an fMRI paradigm suitable for assessing activation in the core system of face processing in young children at the single subject level. Second, we planned to collect data for a power analysis to calculate the necessary group size for a large-scale cross-sectional imaging study assessing the ontogenetic development of the lateralization of the face processing network, with focus on the FFA. It was possible to detect brain activity in the core system of 75% of children at the single subject level. The average scan-to-scan motion of the included children was comparable to adults, ruling out that potential activation differences between groups are caused by unequal motion artifacts. Hemispheric lateralization of the FFA was 0.07 ± 0.48 in children (indicating bilateral activation) and −0.32 ± 0.52 in adults (indicating right-hemispheric dominance). These results thus showed, as expected, a trend for increased lateralization in adults. The estimated effect size for the FFA lateralization difference was d = 0.78 (indicating medium to large effects). An adequately powered follow-up study (sensitivity 0.8) testing developmental changes of FFA lateralization would therefore require the inclusion of 18 children and 26 adults.
Collapse
Affiliation(s)
- Franziska E Hildesheim
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany
| | - Isabell Debus
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany
| | - Roman Kessler
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany
| | - Ina Thome
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany
| | - Kristin M Zimmermann
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany
| | - Olaf Steinsträter
- Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany.,Core-Facility Brainimaging, Faculty of Medicine, Philipps-University Marburg, Marburg, Germany
| | - Jens Sommer
- Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany.,Core-Facility Brainimaging, Faculty of Medicine, Philipps-University Marburg, Marburg, Germany
| | - Inge Kamp-Becker
- Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany.,Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Philipps-University Marburg, Marburg, Germany
| | - Rudolf Stark
- Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany.,Bender Institute of Neuroimaging, Justus-Liebig University Giessen, Giessen, Germany
| | - Andreas Jansen
- Laboratory for Multimodal Neuroimaging, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany.,Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg and Justus-Liebig University Giessen, Giessen, Germany.,Core-Facility Brainimaging, Faculty of Medicine, Philipps-University Marburg, Marburg, Germany
| |
Collapse
|
24
|
Sliwinska MW, Bearpark C, Corkhill J, McPhillips A, Pitcher D. Dissociable pathways for moving and static face perception begin in early visual cortex: Evidence from an acquired prosopagnosic. Cortex 2020; 130:327-339. [DOI: 10.1016/j.cortex.2020.03.033] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2019] [Revised: 02/14/2020] [Accepted: 03/13/2020] [Indexed: 11/25/2022]
|
25
|
Puckett AM, Schira MM, Isherwood ZJ, Victor JD, Roberts JA, Breakspear M. Manipulating the structure of natural scenes using wavelets to study the functional architecture of perceptual hierarchies in the brain. Neuroimage 2020; 221:117173. [PMID: 32682991 PMCID: PMC8239382 DOI: 10.1016/j.neuroimage.2020.117173] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2019] [Revised: 05/11/2020] [Accepted: 07/14/2020] [Indexed: 01/08/2023] Open
Abstract
Functional neuroimaging experiments that employ naturalistic stimuli (natural scenes, films, spoken narratives) provide insights into cognitive function "in the wild". Natural stimuli typically possess crowded, spectrally dense, dynamic, and multimodal properties within a rich multiscale structure. However, when using natural stimuli, various challenges exist for creating parametric manipulations with tight experimental control. Here, we revisit the typical spectral composition and statistical dependences of natural scenes, which distinguish them from abstract stimuli. We then demonstrate how to selectively degrade subtle statistical dependences within specific spatial scales using the wavelet transform. Such manipulations leave basic features of the stimuli, such as luminance and contrast, intact. Using functional neuroimaging of human participants viewing degraded natural images, we demonstrate that cortical responses at different levels of the visual hierarchy are differentially sensitive to subtle statistical dependences in natural images. This demonstration supports the notion that perceptual systems in the brain are optimally tuned to the complex statistical properties of the natural world. The code to undertake these stimulus manipulations, and their natural extension to dynamic natural scenes (films), is freely available.
Collapse
Affiliation(s)
- Alexander M Puckett
- School of Psychology, The University of Queensland, Brisbane QLD 4072, Australia; Queensland Brain Institute, The University of Queensland, Brisbane QLD 4072, Australia.
| | - Mark M Schira
- School of Psychology, University of Wollongong, Wollongong NSW 2522, Australia
| | - Zoey J Isherwood
- School of Psychology, University of Nevada, Reno NV 89557, United States
| | - Jonathan D Victor
- Feil Family Brain and Mind Research Institute and Department of Neurology, Weill Cornell Medical College, New York NY 10065, United States
| | - James A Roberts
- Brain Modelling Group, QIMR Berghofer Medical Research Institute, Brisbane QLD 4006, Australia
| | - Michael Breakspear
- Brain and Mind PRC, University of Newcastle, Newcastle NSW 2308, Australia
| |
Collapse
|
26
|
Williams JHG, Huggins CF, Zupan B, Willis M, Van Rheenen TE, Sato W, Palermo R, Ortner C, Krippl M, Kret M, Dickson JM, Li CSR, Lowe L. A sensorimotor control framework for understanding emotional communication and regulation. Neurosci Biobehav Rev 2020; 112:503-518. [PMID: 32070695 PMCID: PMC7505116 DOI: 10.1016/j.neubiorev.2020.02.014] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2018] [Revised: 01/22/2020] [Accepted: 02/11/2020] [Indexed: 12/12/2022]
Abstract
Our research team was asked to consider the relationship of the neuroscience of sensorimotor control to the language of emotions and feelings. Actions are the principal means for the communication of emotions and feelings in both humans and other animals, and the allostatic mechanisms controlling action also apply to the regulation of emotional states by the self and others. We consider how motor control of hierarchically organised, feedback-based, goal-directed action has evolved in humans, within a context of consciousness, appraisal and cultural learning, to serve emotions and feelings. In our linguistic analysis, we found that many emotion and feelings words could be assigned to stages in the sensorimotor learning process, but the assignment was often arbitrary. The embodied nature of emotional communication means that action words are frequently used, but that the meanings or senses of the word depend on its contextual use, just as the relationship of an action to an emotion is also contextually dependent.
Collapse
Affiliation(s)
- Justin H G Williams
- University of Aberdeen, Institute of Medical Sciences, Foresterhill, AB25 2ZD, Scotland, United Kingdom.
| | - Charlotte F Huggins
- University of Aberdeen, Institute of Medical Sciences, Foresterhill, AB25 2ZD, Scotland, United Kingdom
| | - Barbra Zupan
- Central Queensland University, School of Health, Medical and Applied Sciences, Bruce Highway, Rockhampton, QLD 4702, Australia
| | - Megan Willis
- Australian Catholic University, School of Psychology, ARC Centre for Excellence in Cognition and its Disorders, Sydney, NSW 2060, Australia
| | - Tamsyn E Van Rheenen
- University of Melbourne, Melbourne Neuropsychiatry Centre, Department of Psychiatry, 161 Barry Street, Carlton, VIC 3053, Australia
| | - Wataru Sato
- Kyoto University, Kokoro Research Centre, 46 Yoshidashimoadachicho, Sakyo Ward, Kyoto, 606-8501, Japan
| | - Romina Palermo
- University of Western Australia, School of Psychological Science, Perth, WA, 6009, Australia
| | - Catherine Ortner
- Thompson Rivers University, Department of Psychology, 805 TRU Way, Kamloops, BC V2C 0C8, Canada
| | - Martin Krippl
- Otto von Guericke University Magdeburg, Faculty of Natural Sciences, Department of Psychology, Universitätsplatz 2, Magdeburg, 39106, Germany
| | - Mariska Kret
- Leiden University, Cognitive Psychology, Pieter de la Court, Waassenaarseweg 52, Leiden, 2333 AK, the Netherlands
| | - Joanne M Dickson
- Edith Cowan University, Psychology Department, School of Arts and Humanities, 270 Joondalup Dr, Joondalup, WA 6027, Australia
| | - Chiang-Shan R Li
- Yale University, Connecticut Mental Health Centre, S112, 34 Park Street, New Haven, CT 06519-1109, USA
| | - Leroy Lowe
- Neuroqualia, Room 229A, Forrester Hall, 36 Arthur Street, Truro, Nova Scotia, B2N 1X5, Canada
| |
Collapse
|
27
|
Zhang H, Japee S, Stacy A, Flessert M, Ungerleider LG. Anterior superior temporal sulcus is specialized for non-rigid facial motion in both monkeys and humans. Neuroimage 2020; 218:116878. [PMID: 32360168 PMCID: PMC7478875 DOI: 10.1016/j.neuroimage.2020.116878] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2019] [Revised: 04/18/2020] [Accepted: 04/23/2020] [Indexed: 02/01/2023] Open
Abstract
Facial motion plays a fundamental role in the recognition of facial expressions in primates, but the neural substrates underlying this special type of biological motion are not well understood. Here, we used fMRI to investigate the extent to which the specialization for facial motion is represented in the visual system and compared the neural mechanisms for the processing of non-rigid facial motion in macaque monkeys and humans. We defined the areas specialized for facial motion as those significantly more activated when subjects perceived the motion caused by dynamic faces (dynamic faces > static faces) than when they perceived the motion caused by dynamic non-face objects (dynamic objects > static objects). We found that, in monkeys, significant activations evoked by facial motion were in the fundus of anterior superior temporal sulcus (STS), which overlapped the anterior fundus face patch. In humans, facial motion activated three separate foci in the right STS: posterior, middle, and anterior STS, with the anterior STS location showing the most selectivity for facial motion compared with other facial motion areas. In both monkeys and humans, facial motion shows a gradient preference as one progresses anteriorly along the STS. Taken together, our results indicate that monkeys and humans share similar neural substrates within the anterior temporal lobe specialized for the processing of non-rigid facial motion.
Collapse
Affiliation(s)
- Hui Zhang
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, MD, 20892, USA; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing, 100191, China.
| | - Shruti Japee
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, MD, 20892, USA
| | - Andrea Stacy
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, MD, 20892, USA
| | - Molly Flessert
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, MD, 20892, USA
| | | |
Collapse
|
28
|
Borowiak K, Maguinness C, von Kriegstein K. Dorsal-movement and ventral-form regions are functionally connected during visual-speech recognition. Hum Brain Mapp 2020; 41:952-972. [PMID: 31749219 PMCID: PMC7267922 DOI: 10.1002/hbm.24852] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2019] [Revised: 09/03/2019] [Accepted: 10/21/2019] [Indexed: 01/17/2023] Open
Abstract
Faces convey social information such as emotion and speech. Facial emotion processing is supported via interactions between dorsal-movement and ventral-form visual cortex regions. Here, we explored, for the first time, whether similar dorsal-ventral interactions (assessed via functional connectivity), might also exist for visual-speech processing. We then examined whether altered dorsal-ventral connectivity is observed in adults with high-functioning autism spectrum disorder (ASD), a disorder associated with impaired visual-speech recognition. We acquired functional magnetic resonance imaging (fMRI) data with concurrent eye tracking in pairwise matched control and ASD participants. In both groups, dorsal-movement regions in the visual motion area 5 (V5/MT) and the temporal visual speech area (TVSA) were functionally connected to ventral-form regions (i.e., the occipital face area [OFA] and the fusiform face area [FFA]) during the recognition of visual speech, in contrast to the recognition of face identity. Notably, parts of this functional connectivity were decreased in the ASD group compared to the controls (i.e., right V5/MT-right OFA, left TVSA-left FFA). The results confirmed our hypothesis that functional connectivity between dorsal-movement and ventral-form regions exists during visual-speech processing. Its partial dysfunction in ASD might contribute to difficulties in the recognition of dynamic face information relevant for successful face-to-face communication.
Collapse
Affiliation(s)
- Kamila Borowiak
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
- Berlin School of Mind and Brain, Humboldt University of BerlinBerlinGermany
| | - Corrina Maguinness
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| | - Katharina von Kriegstein
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| |
Collapse
|
29
|
Kätsyri J, de Gelder B, de Borst AW. Amygdala responds to direct gaze in real but not in computer-generated faces. Neuroimage 2019; 204:116216. [PMID: 31553928 DOI: 10.1016/j.neuroimage.2019.116216] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 08/22/2019] [Accepted: 09/19/2019] [Indexed: 10/25/2022] Open
Abstract
Computer-generated (CG) faces are an important visual interface for human-computer interaction in social contexts. Here we investigated whether the human brain processes emotion and gaze similarly in real and carefully matched CG faces. Real faces evoked greater responses in the fusiform face area than CG faces, particularly for fearful expressions. Emotional (angry and fearful) facial expressions evoked similar activations in the amygdala in real and CG faces. Direct as compared with averted gaze elicited greater fMRI responses in the amygdala regardless of facial expression but only for real and not for CG faces. We observed an interaction effect between gaze and emotion (i.e., the shared signal effect) in the right posterior temporal sulcus and other regions, but not in the amygdala, and we found no evidence for different shared signal effects in real and CG faces. Taken together, the present findings highlight similarities (emotional processing in the amygdala) and differences (overall processing in the fusiform face area, gaze processing in the amygdala) in the neural processing of real and CG faces.
Collapse
Affiliation(s)
- Jari Kätsyri
- Brain and Emotion Laboratory, Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, the Netherlands; Department of Computer Science, Aalto University, Espoo, Finland.
| | - Beatrice de Gelder
- Brain and Emotion Laboratory, Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, the Netherlands; Department of Computer Science, University College London, London, United Kingdom
| | - Aline W de Borst
- UCL Interaction Centre, University College London, London, United Kingdom
| |
Collapse
|
30
|
Schultz J, Willems T, Gädeke M, Chakkour G, Franke A, Weber B, Hurlemann R. A human subcortical network underlying social avoidance revealed by risky economic choices. eLife 2019; 8:45249. [PMID: 31329098 PMCID: PMC6703852 DOI: 10.7554/elife.45249] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2019] [Accepted: 07/21/2019] [Indexed: 12/02/2022] Open
Abstract
Social interactions have a major impact on well-being. While many individuals actively seek social situations, others avoid them, at great cost to their private and professional life. The neural mechanisms underlying individual differences in social approach or avoidance tendencies are poorly understood. Here we estimated people’s subjective value of engaging in a social situation. In each trial, more or less socially anxious participants chose between an interaction with a human partner providing social feedback and a monetary amount. With increasing social anxiety, the subjective value of social engagement decreased; amygdala BOLD response during decision-making and when experiencing social feedback increased; ventral striatum BOLD response to positive social feedback decreased; and connectivity between these regions during decision-making increased. Amygdala response was negatively related to the subjective value of social engagement. These findings suggest a relation between trait social anxiety/social avoidance and activity in a subcortical network during social decision-making. Your relationships with the people around you – friends, family, colleagues – have a strong influence on your overall life happiness. Even so, many people struggle to engage with the people around them. Social interactions can be stressful and many people choose to avoid them, even at a cost. Being able to measure these tendencies experimentally is a first useful step for assessing social avoidance without relying on people’s, often biased, recollections of their actions and behaviours. But how can a tendency to avoid social situations be quantified? And what can an experiment to measure this tendency reveal about the neural underpinnings of social avoidance? Schultz et al. asked volunteers to play a social game. If they played, the volunteers had the chance to win three euros, but they could choose not to play and receive a fixed amount of money, which varied across trials between zero and three euros. This approach allowed Schultz et al. to quantify how much the volunteers valued playing the game. The game involved playing with other virtual human partners, who gave either positive or negative social feedback depending on the outcome of the game in the form of videos of facial expressions. In a non-social control experiment, a computer gave abstract feedback in the form of symbols. Schultz et al. found that the value people placed on playing the social game varied with their level of social anxiety (established using a standard questionnaire). The more anxious people attributed less value to engaging in the game. Neuroimaging experiments revealed that the activity and connectivity between the amygdala and ventral striatum, two parts of the brain involved in processing emotions and reward-related stimuli, varied according to people’s levels of social anxiety. Social interactions have a major impact on the quality of life of both healthy people and those with mental disorders. Developing new ways to measure and understand the differences in the brain linked to social traits could help to characterise certain conditions and document therapy progress. Methods to quantify social anxiety and avoidance are also in line with efforts to explore the neuroscience behind the full range of human behaviour.
Collapse
Affiliation(s)
- Johannes Schultz
- Division of Medical Psychology, University of Bonn, Bonn, Germany.,Center for Economics and Neuroscience, University of Bonn, Bonn, Germany.,Institute of Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany
| | - Tom Willems
- Division of Medical Psychology, University of Bonn, Bonn, Germany
| | - Maria Gädeke
- Division of Medical Psychology, University of Bonn, Bonn, Germany
| | - Ghada Chakkour
- Division of Medical Psychology, University of Bonn, Bonn, Germany.,Medical School, University of Bonn, Bonn, Germany
| | - Alexander Franke
- Division of Medical Psychology, University of Bonn, Bonn, Germany.,Medical School, University of Bonn, Bonn, Germany
| | - Bernd Weber
- Center for Economics and Neuroscience, University of Bonn, Bonn, Germany.,Institute of Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany
| | - Rene Hurlemann
- Division of Medical Psychology, University of Bonn, Bonn, Germany.,Department of Psychiatry and Psychotherapy, University of Bonn, Bonn, Germany
| |
Collapse
|
31
|
Plouffe-Demers MP, Fiset D, Saumure C, Duncan J, Blais C. Strategy Shift Toward Lower Spatial Frequencies in the Recognition of Dynamic Facial Expressions of Basic Emotions: When It Moves It Is Different. Front Psychol 2019; 10:1563. [PMID: 31379648 PMCID: PMC6650765 DOI: 10.3389/fpsyg.2019.01563] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2018] [Accepted: 06/20/2019] [Indexed: 11/15/2022] Open
Abstract
Facial expressions of emotion play a key role in social interactions. While in everyday life, their dynamic and transient nature calls for a fast processing of the visual information they contain, a majority of studies investigating the visual processes underlying their recognition have focused on their static display. The present study aimed to gain a better understanding of these processes while using more ecological dynamic facial expressions. In two experiments, we directly compared the spatial frequency (SF) tuning during the recognition of static and dynamic facial expressions. Experiment 1 revealed a shift toward lower SFs for dynamic expressions in comparison to static ones. Experiment 2 was designed to verify if changes in SF tuning curves were specific to the presence of emotional information in motion by comparing the SF tuning profiles for static, dynamic, and shuffled dynamic expressions. Results showed a similar shift toward lower SFs for shuffled expressions, suggesting that the difference found between dynamic and static expressions might not be linked to informative motion per se but to the presence of motion regardless its nature.
Collapse
Affiliation(s)
- Marie-Pier Plouffe-Demers
- Département de Psychologie, Universtité du Québec en Outaouais, Gatineau, QC, Canada
- Département de Psychologie, Université du Québec à Montréal, Montreal, QC, Canada
| | - Daniel Fiset
- Département de Psychologie, Universtité du Québec en Outaouais, Gatineau, QC, Canada
| | - Camille Saumure
- Département de Psychologie, Universtité du Québec en Outaouais, Gatineau, QC, Canada
| | - Justin Duncan
- Département de Psychologie, Universtité du Québec en Outaouais, Gatineau, QC, Canada
- Département de Psychologie, Université du Québec à Montréal, Montreal, QC, Canada
| | - Caroline Blais
- Département de Psychologie, Universtité du Québec en Outaouais, Gatineau, QC, Canada
| |
Collapse
|
32
|
Kikuchi Y, Ip J, Lagier G, Mossom JC, Kumar S, Petkov CI, Barraclough NE, Vuong QC. Interactions between Conscious and Subconscious Signals: Selective Attention under Feature-Based Competition Increases Neural Selectivity during Brain Adaptation. J Neurosci 2019; 39:5506-5516. [PMID: 31068438 PMCID: PMC6616293 DOI: 10.1523/jneurosci.3052-18.2019] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2018] [Revised: 03/14/2019] [Accepted: 04/06/2019] [Indexed: 01/08/2023] Open
Abstract
Efficient perception in natural environments depends on neural interactions between voluntary processes within cognitive control, such as attention, and those that are automatic and subconscious, such as brain adaptation to predictable input (also called repetition suppression). Although both attention and adaptation have been studied separately and there is considerable knowledge of the neurobiology involved in each of these processes, how attention interacts with adaptation remains equivocal. We examined how attention interacts with visual and auditory adaptation by measuring neuroimaging effects consistent with changes in either neural gain or selectivity. Male and female human participants were scanned with functional magnetic resonance imaging (fMRI) first while they discriminated repetition of morphed faces or voices and either directed their attention to stimulus identity or spatial location. Attention to face or voice identity, while ignoring stimulus location, solely increased the gain of respectively face- or voice-sensitive cortex. The results were strikingly different in an experiment when participants attended to voice identity versus stimulus loudness. In this case, attention to voice while ignoring sound loudness increased neural selectivity. The combined results show that how attention affects adaptation depends on the level of feature-based competition, reconciling prior conflicting observations. The findings are theoretically important and are discussed in relation to neurobiological interactions between attention and different types of predictive signals.SIGNIFICANCE STATEMENT Adaptation to repeated environmental events is ubiquitous in the animal brain, an automatic typically subconscious, predictive signal. Cognitive influences, such as by attention, powerfully affect sensory processing and can overcome brain adaptation. However, how neural interactions occur between adaptation and attention remains controversial. We conducted fMRI experiments regulating the focus of attention during adaptation to repeated stimuli with perceptually balanced stimulus expectancy. We observed an interaction between attention and adaptation consistent with increased neural selectivity, but only under conditions of feature-based competition, challenging the notion that attention interacts with brain adaptation by only affecting response gain. This demonstrates that attention retains its full complement of mechanistic influences on sensory cortex even as it interacts with more automatic or subconscious predictive processes.
Collapse
Affiliation(s)
- Yukiko Kikuchi
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - Jennifer Ip
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - Gaëtan Lagier
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - James C Mossom
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - Sukhbinder Kumar
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - Christopher I Petkov
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| | - Nick E Barraclough
- Department of Psychology, University of York, Heslington, York YO10 5DD, United Kingdom
| | - Quoc C Vuong
- Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne NE2 4HH, United Kingdom, and
| |
Collapse
|
33
|
Naturalistic Stimuli in Neuroscience: Critically Acclaimed. Trends Cogn Sci 2019; 23:699-714. [PMID: 31257145 DOI: 10.1016/j.tics.2019.05.004] [Citation(s) in RCA: 232] [Impact Index Per Article: 46.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2019] [Revised: 05/08/2019] [Accepted: 05/21/2019] [Indexed: 01/12/2023]
Abstract
Cognitive neuroscience has traditionally focused on simple tasks, presented sparsely and using abstract stimuli. While this approach has yielded fundamental insights into functional specialisation in the brain, its ecological validity remains uncertain. Do these tasks capture how brains function 'in the wild', where stimuli are dynamic, multimodal, and crowded? Ecologically valid paradigms that approximate real life scenarios, using stimuli such as films, spoken narratives, music, and multiperson games emerged in response to these concerns over a decade ago. We critically appraise whether this approach has delivered on its promise to deliver new insights into brain function. We highlight the challenges, technological innovations, and clinical opportunities that are required should this field meet its full potential.
Collapse
|
34
|
Pitcher D, Ianni G, Ungerleider LG. A functional dissociation of face-, body- and scene-selective brain areas based on their response to moving and static stimuli. Sci Rep 2019; 9:8242. [PMID: 31160680 PMCID: PMC6546694 DOI: 10.1038/s41598-019-44663-9] [Citation(s) in RCA: 37] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2018] [Accepted: 05/07/2019] [Indexed: 11/20/2022] Open
Abstract
The human brain contains areas that respond selectively to faces, bodies and scenes. Neuroimaging studies have shown that a subset of these areas preferentially respond more to moving than static stimuli, but the reasons for this functional dissociation remain unclear. In the present study, we simultaneously mapped the responses to motion in face-, body- and scene-selective areas in the right hemisphere using moving and static stimuli. Participants (N = 22) were scanned using functional magnetic resonance imaging (fMRI) while viewing videos containing bodies, faces, objects, scenes or scrambled objects, and static pictures from the beginning, middle and end of each video. Results demonstrated that lateral areas, including face-selective areas in the posterior and anterior superior temporal sulcus (STS), the extrastriate body area (EBA) and the occipital place area (OPA) responded more to moving than static stimuli. By contrast, there was no difference between the response to moving and static stimuli in ventral and medial category-selective areas, including the fusiform face area (FFA), occipital face area (OFA), amygdala, fusiform body area (FBA), retrosplenial complex (RSC) and parahippocampal place area (PPA). This functional dissociation between lateral and ventral/medial brain areas that respond selectively to different visual categories suggests that face-, body- and scene-selective networks may be functionally organized along a common dimension.
Collapse
Affiliation(s)
- David Pitcher
- Department of Psychology, University of York, Heslington, York, YO105DD, UK.
| | - Geena Ianni
- Weill Cornell/Rockefeller/Sloan Kettering Tri-Institutional MD-PhD Program, New York, 10065, USA.,Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD, 20892, USA
| | - Leslie G Ungerleider
- Section on Neurocircuitry, Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD, 20892, USA
| |
Collapse
|
35
|
Sato W, Kochiyama T, Uono S, Sawada R, Kubota Y, Yoshimura S, Toichi M. Widespread and lateralized social brain activity for processing dynamic facial expressions. Hum Brain Mapp 2019; 40:3753-3768. [PMID: 31090126 DOI: 10.1002/hbm.24629] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2019] [Revised: 04/28/2019] [Accepted: 05/02/2019] [Indexed: 11/07/2022] Open
Abstract
Dynamic facial expressions of emotions constitute natural and powerful means of social communication in daily life. A number of previous neuroimaging studies have explored the neural mechanisms underlying the processing of dynamic facial expressions, and indicated the activation of certain social brain regions (e.g., the amygdala) during such tasks. However, the activated brain regions were inconsistent across studies, and their laterality was rarely evaluated. To investigate these issues, we measured brain activity using functional magnetic resonance imaging in a relatively large sample (n = 51) during the observation of dynamic facial expressions of anger and happiness and their corresponding dynamic mosaic images. The observation of dynamic facial expressions, compared with dynamic mosaics, elicited stronger activity in the bilateral posterior cortices, including the inferior occipital gyri, fusiform gyri, and superior temporal sulci. The dynamic facial expressions also activated bilateral limbic regions, including the amygdalae and ventromedial prefrontal cortices, more strongly versus mosaics. In the same manner, activation was found in the right inferior frontal gyrus (IFG) and left cerebellum. Laterality analyses comparing original and flipped images revealed right hemispheric dominance in the superior temporal sulcus and IFG and left hemispheric dominance in the cerebellum. These results indicated that the neural mechanisms underlying processing of dynamic facial expressions include widespread social brain regions associated with perceptual, emotional, and motor functions, and include a clearly lateralized (right cortical and left cerebellar) network like that involved in language processing.
Collapse
Affiliation(s)
- Wataru Sato
- Kokoro Research Center, Kyoto University, Kyoto, Japan
| | | | - Shota Uono
- Department of Neurodevelopmental Psychiatry, Habilitation and Rehabilitation, Kyoto University, Kyoto, Japan
| | - Reiko Sawada
- Department of Neurodevelopmental Psychiatry, Habilitation and Rehabilitation, Kyoto University, Kyoto, Japan
| | - Yasutaka Kubota
- Health and Medical Services Center, Shiga University, Hikone, Shiga, Japan
| | - Sayaka Yoshimura
- Department of Neurodevelopmental Psychiatry, Habilitation and Rehabilitation, Kyoto University, Kyoto, Japan
| | - Motomi Toichi
- Faculty of Human Health Science, Kyoto University, Kyoto, Japan.,The Organization for Promoting Neurodevelopmental Disorder Research, Kyoto, Japan
| |
Collapse
|
36
|
Pathways for smiling, disgust and fear recognition in blindsight patients. Neuropsychologia 2019; 128:6-13. [DOI: 10.1016/j.neuropsychologia.2017.08.028] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2017] [Revised: 08/03/2017] [Accepted: 08/28/2017] [Indexed: 01/08/2023]
|
37
|
Vanderwal T, Eilbott J, Castellanos FX. Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Dev Cogn Neurosci 2019; 36:100600. [PMID: 30551970 PMCID: PMC6969259 DOI: 10.1016/j.dcn.2018.10.004] [Citation(s) in RCA: 96] [Impact Index Per Article: 19.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2018] [Revised: 09/13/2018] [Accepted: 10/08/2018] [Indexed: 11/28/2022] Open
Abstract
The use of movie-watching as an acquisition state for functional connectivity (FC) MRI has recently enabled multiple groups to obtain rich data sets in younger children with both substantial sample sizes and scan durations. Using naturalistic paradigms such as movies has also provided analytic flexibility for these developmental studies that extends beyond conventional resting state approaches. This review highlights the advantages and challenges of using movies for developmental neuroimaging and explores some of the methodological issues involved in designing pediatric studies with movies. Emerging themes from movie-watching studies are discussed, including an emphasis on intersubject correlations, developmental changes in network interactions under complex naturalistic conditions, and dynamic age-related changes in both sensory and higher-order network FC even in narrow age ranges. Converging evidence suggests an enhanced ability to identify brain-behavior correlations in children when using movie-watching data relative to both resting state and conventional tasks. Future directions and cautionary notes highlight the potential and the limitations of using movies to study FC in pediatric populations.
Collapse
Affiliation(s)
- Tamara Vanderwal
- University of British Columbia, 2255 Wesbrook Mall, Vancouver, BC, V6T 2A1, Canada; Yale Child Study Center, 230 South Frontage Road, New Haven CT, 06519, United States.
| | - Jeffrey Eilbott
- Yale Child Study Center, 230 South Frontage Road, New Haven CT, 06519, United States
| | - F Xavier Castellanos
- The Child Study Center at New York University Langone Medical Center, 1 Park Avenue, New York, NY, 10016, United States; Nathan Kline Institute for Psychiatric Research, 140 Old Orangeburg Road, Orangeburg, NY, 10962, United States
| |
Collapse
|
38
|
Rymarczyk K, Żurawski Ł, Jankowiak-Siuda K, Szatkowska I. Empathy in Facial Mimicry of Fear and Disgust: Simultaneous EMG-fMRI Recordings During Observation of Static and Dynamic Facial Expressions. Front Psychol 2019; 10:701. [PMID: 30971997 PMCID: PMC6445885 DOI: 10.3389/fpsyg.2019.00701] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2018] [Accepted: 03/13/2019] [Indexed: 01/18/2023] Open
Abstract
Real-life faces are dynamic by nature, particularly when expressing emotion. Increasing evidence suggests that the perception of dynamic displays enhances facial mimicry and induces activation in widespread brain structures considered to be part of the mirror neuron system, a neuronal network linked to empathy. The present study is the first to investigate the relations among facial muscle responses, brain activity, and empathy traits while participants observed static and dynamic (videos) facial expressions of fear and disgust. During display presentation, blood-oxygen level-dependent (BOLD) signal as well as muscle reactions of the corrugator supercilii and levator labii were recorded simultaneously from 46 healthy individuals (21 females). It was shown that both fear and disgust faces caused activity in the corrugator supercilii muscle, while perception of disgust produced facial activity additionally in the levator labii muscle, supporting a specific pattern of facial mimicry for these emotions. Moreover, individuals with higher, compared to individuals with lower, empathy traits showed greater activity in the corrugator supercilii and levator labii muscles; however, these responses were not differentiable between static and dynamic mode. Conversely, neuroimaging data revealed motion and emotional-related brain structures in response to dynamic rather than static stimuli among high empathy individuals. In line with this, there was a correlation between electromyography (EMG) responses and brain activity suggesting that the Mirror Neuron System, the anterior insula and the amygdala might constitute the neural correlates of automatic facial mimicry for fear and disgust. These results revealed that the dynamic property of (emotional) stimuli facilitates the emotional-related processing of facial expressions, especially among whose with high trait empathy.
Collapse
Affiliation(s)
- Krystyna Rymarczyk
- Department of Experimental Psychology, Institute of Cognitive and Behavioural Neuroscience, SWPS University of Social Sciences and Humanities, Warsaw, Poland
| | - Łukasz Żurawski
- Laboratory of Psychophysiology, Department of Neurophysiology, Nencki Institute of Experimental Biology, Polish Academy of Sciences (PAS), Warsaw, Poland
| | - Kamila Jankowiak-Siuda
- Department of Experimental Psychology, Institute of Cognitive and Behavioural Neuroscience, SWPS University of Social Sciences and Humanities, Warsaw, Poland
| | - Iwona Szatkowska
- Laboratory of Psychophysiology, Department of Neurophysiology, Nencki Institute of Experimental Biology, Polish Academy of Sciences (PAS), Warsaw, Poland
| |
Collapse
|
39
|
Calvo MG, Fernández-Martín A, Gutiérrez-García A, Lundqvist D. Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database. Sci Rep 2018; 8:17039. [PMID: 30451919 PMCID: PMC6242984 DOI: 10.1038/s41598-018-35259-w] [Citation(s) in RCA: 32] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2018] [Accepted: 10/28/2018] [Indexed: 12/20/2022] Open
Abstract
Prior research using static facial stimuli (photographs) has identified diagnostic face regions (i.e., functional for recognition) of emotional expressions. In the current study, we aimed to determine attentional orienting, engagement, and time course of fixation on diagnostic regions. To this end, we assessed the eye movements of observers inspecting dynamic expressions that changed from a neutral to an emotional face. A new stimulus set (KDEF-dyn) was developed, which comprises 240 video-clips of 40 human models portraying six basic emotions (happy, sad, angry, fearful, disgusted, and surprised). For validation purposes, 72 observers categorized the expressions while gaze behavior was measured (probability of first fixation, entry time, gaze duration, and number of fixations). Specific visual scanpath profiles characterized each emotional expression: The eye region was looked at earlier and longer for angry and sad faces; the mouth region, for happy faces; and the nose/cheek region, for disgusted faces; the eye and the mouth regions attracted attention in a more balanced manner for surprise and fear. These profiles reflected enhanced selective attention to expression-specific diagnostic face regions. The KDEF-dyn stimuli and the validation data will be available to the scientific community as a useful tool for research on emotional facial expression processing.
Collapse
Affiliation(s)
- Manuel G Calvo
- Department of Cognitive Psychology, Universidad de La Laguna, Tenerife, Spain.
- Instituto Universitario de Neurociencia (IUNE), Universidad de La Laguna, Tenerife, Spain.
| | | | | | | |
Collapse
|
40
|
Calvo MG, Fernández-Martín A, Recio G, Lundqvist D. Human Observers and Automated Assessment of Dynamic Emotional Facial Expressions: KDEF-dyn Database Validation. Front Psychol 2018; 9:2052. [PMID: 30416473 PMCID: PMC6212581 DOI: 10.3389/fpsyg.2018.02052] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2018] [Accepted: 10/05/2018] [Indexed: 12/11/2022] Open
Abstract
Most experimental studies of facial expression processing have used static stimuli (photographs), yet facial expressions in daily life are generally dynamic. In its original photographic format, the Karolinska Directed Emotional Faces (KDEF) has been frequently utilized. In the current study, we validate a dynamic version of this database, the KDEF-dyn. To this end, we applied animation between neutral and emotional expressions (happy, sad, angry, fearful, disgusted, and surprised; 1,033-ms unfolding) to 40 KDEF models, with morphing software. Ninety-six human observers categorized the expressions of the resulting 240 video-clip stimuli, and automated face analysis assessed the evidence for 6 expressions and 20 facial action units (AUs) at 31 intensities. Low-level image properties (luminance, signal-to-noise ratio, etc.) and other purely perceptual factors (e.g., size, unfolding speed) were controlled. Human recognition performance (accuracy, efficiency, and confusions) patterns were consistent with prior research using static and other dynamic expressions. Automated assessment of expressions and AUs was sensitive to intensity manipulations. Significant correlations emerged between human observers' categorization and automated classification. The KDEF-dyn database aims to provide a balance between experimental control and ecological validity for research on emotional facial expression processing. The stimuli and the validation data are available to the scientific community.
Collapse
Affiliation(s)
- Manuel G. Calvo
- Department of Cognitive Psychology, Universidad de La Laguna, San Cristóbal de La Laguna, Spain
- Instituto Universitario de Neurociencia (IUNE), Universidad de La Laguna, Santa Cruz de Tenerife, Spain
| | | | - Guillermo Recio
- Institute of Psychology, Universität Hamburg, Hamburg, Germany
| | - Daniel Lundqvist
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
41
|
Richoz AR, Lao J, Pascalis O, Caldara R. Tracking the recognition of static and dynamic facial expressions of emotion across the life span. J Vis 2018; 18:5. [PMID: 30208425 DOI: 10.1167/18.9.5] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
The effective transmission and decoding of dynamic facial expressions of emotion is omnipresent and critical for adapted social interactions in everyday life. Thus, common intuition would suggest an advantage for dynamic facial expression recognition (FER) over the static snapshots routinely used in most experiments. However, although many studies reported an advantage in the recognition of dynamic over static expressions in clinical populations, results obtained from healthy participants are contrasted. To clarify this issue, we conducted a large cross-sectional study to investigate FER across the life span in order to determine if age is a critical factor to account for such discrepancies. More than 400 observers (age range 5-96) performed recognition tasks of the six basic expressions in static, dynamic, and shuffled (temporally randomized frames) conditions, normalized for the amount of energy sampled over time. We applied a Bayesian hierarchical step-linear model to capture the nonlinear relationship between age and FER for the different viewing conditions. Although replicating the typical accuracy profiles of FER, we determined the age at which peak efficiency was reached for each expression and found greater accuracy for most dynamic expressions across the life span. This advantage in the elderly population was driven by a significant decrease in performance for static images, which was twice as large as for the young adults. Our data posit the use of dynamic stimuli as being critical in the assessment of FER in the elderly population, inviting caution when drawing conclusions from the sole use of static face images to this aim.
Collapse
Affiliation(s)
- Anne-Raphaëlle Richoz
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland.,LPNC, University of Grenoble Alpes, Grenoble, France
| | - Junpeng Lao
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | | | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| |
Collapse
|
42
|
Dobs K, Bülthoff I, Schultz J. Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies-a Review of Behavioral Findings and Methodology. Front Psychol 2018; 9:1355. [PMID: 30123162 PMCID: PMC6085596 DOI: 10.3389/fpsyg.2018.01355] [Citation(s) in RCA: 40] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2018] [Accepted: 07/13/2018] [Indexed: 01/01/2023] Open
Abstract
Faces that move contain rich information about facial form, such as facial features and their configuration, alongside the motion of those features. During social interactions, humans constantly decode and integrate these cues. To fully understand human face perception, it is important to investigate what information dynamic faces convey and how the human visual system extracts and processes information from this visual input. However, partly due to the difficulty of designing well-controlled dynamic face stimuli, many face perception studies still rely on static faces as stimuli. Here, we focus on evidence demonstrating the usefulness of dynamic faces as stimuli, and evaluate different types of dynamic face stimuli to study face perception. Studies based on dynamic face stimuli revealed a high sensitivity of the human visual system to natural facial motion and consistently reported dynamic advantages when static face information is insufficient for the task. These findings support the hypothesis that the human perceptual system integrates sensory cues for robust perception. In the present paper, we review the different types of dynamic face stimuli used in these studies, and assess their usefulness for several research questions. Natural videos of faces are ecological stimuli but provide limited control of facial form and motion. Point-light faces allow for good control of facial motion but are highly unnatural. Image-based morphing is a way to achieve control over facial motion while preserving the natural facial form. Synthetic facial animations allow separation of facial form and motion to study aspects such as identity-from-motion. While synthetic faces are less natural than videos of faces, recent advances in photo-realistic rendering may close this gap and provide naturalistic stimuli with full control over facial motion. We believe that many open questions, such as what dynamic advantages exist beyond emotion and identity recognition and which dynamic aspects drive these advantages, can be addressed adequately with different types of stimuli and will improve our understanding of face perception in more ecological settings.
Collapse
Affiliation(s)
- Katharina Dobs
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, United States.,Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Isabelle Bülthoff
- Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Johannes Schultz
- Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Tübingen, Germany.,Division of Medical Psychology and Department of Psychiatry, University of Bonn, Bonn, Germany
| |
Collapse
|
43
|
Lazzeri N, Mazzei D, Ben Moussa M, Magnenat-Thalmann N, De Rossi D. The influence of dynamics and speech on understanding humanoid facial expressions. INT J ADV ROBOT SYST 2018. [DOI: 10.1177/1729881418783158] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Human communication relies mostly on nonverbal signals expressed through body language. Facial expressions, in particular, convey emotional information that allows people involved in social interactions to mutually judge the emotional states and to adjust its behavior appropriately. First studies aimed at investigating the recognition of facial expressions were based on static stimuli. However, facial expressions are rarely static, especially in everyday social interactions. Therefore, it has been hypothesized that the dynamics inherent in a facial expression could be fundamental in understanding its meaning. In addition, it has been demonstrated that nonlinguistic and linguistic information can contribute to reinforce the meaning of a facial expression making it easier to be recognized. Nevertheless, few studies have been performed on realistic humanoid robots. This experimental work aimed at demonstrating the human-like expressive capability of a humanoid robot by examining whether the effect of motion and vocal content influenced the perception of its facial expressions. The first part of the experiment aimed at studying the recognition capability of two kinds of stimuli related to the six basic expressions (i.e. anger, disgust, fear, happiness, sadness, and surprise): static stimuli, that is, photographs, and dynamic stimuli, that is, video recordings. The second and third parts were focused on comparing the same six basic expressions performed by a virtual avatar and by a physical robot under three different conditions: (1) muted facial expressions, (2) facial expressions with nonlinguistic vocalizations, and (3) facial expressions with an emotionally neutral verbal sentence. The results show that static stimuli performed by a human being and by the robot were more ambiguous than the corresponding dynamic stimuli on which motion and vocalization were associated. This hypothesis has been also investigated with a 3-dimensional replica of the physical robot demonstrating that even in case of a virtual avatar, dynamic and vocalization improve the emotional conveying capability.
Collapse
Affiliation(s)
- Nicole Lazzeri
- Research Center E. Piaggio, Faculty of Engineering, University of Pisa, Pisa, Italy
| | - Daniele Mazzei
- Computer Science Department, University of Pisa, Pisa, Italy
| | - Maher Ben Moussa
- Computer Science Centre, University of Geneva, Geneva, Switzerland
| | | | - Danilo De Rossi
- Research Center E. Piaggio, Faculty of Engineering, University of Pisa, Pisa, Italy
| |
Collapse
|
44
|
Jesse A, Bartoli M. Learning to recognize unfamiliar talkers: Listeners rapidly form representations of facial dynamic signatures. Cognition 2018; 176:195-208. [DOI: 10.1016/j.cognition.2018.03.018] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2017] [Revised: 03/13/2018] [Accepted: 03/21/2018] [Indexed: 11/25/2022]
|
45
|
Nanni M, Martínez-Soto J, Gonzalez-Santos L, Barrios FA. Neural correlates of the natural observation of an emotionally loaded video. PLoS One 2018; 13:e0198731. [PMID: 29883494 PMCID: PMC5993250 DOI: 10.1371/journal.pone.0198731] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2017] [Accepted: 05/24/2018] [Indexed: 01/24/2023] Open
Abstract
Studies based on a paradigm of free or natural viewing have revealed characteristics that allow us to know how the brain processes stimuli within a natural environment. This method has been little used to study brain function. With a connectivity approach, we examine the processing of emotions using an exploratory method to analyze functional magnetic resonance imaging (fMRI) data. This research describes our approach to modeling stress paradigms suitable for neuroimaging environments. We showed a short film (4.54 minutes) with high negative emotional valence and high arousal content to 24 healthy male subjects (36.42 years old; SD = 12.14) during fMRI. Independent component analysis (ICA) was used to identify networks based on spatial statistical independence. Through this analysis we identified the sensorimotor system and its influence on the dorsal attention and default-mode networks, which in turn have reciprocal activity and modulate networks described as emotional.
Collapse
Affiliation(s)
- Melanni Nanni
- Universidad Nacional Autónoma de México, Instituto de Neurobiología, Querétaro, México
| | - Joel Martínez-Soto
- Universidad Nacional Autónoma de México, Instituto de Neurobiología, Querétaro, México
- Department of Psychology, Universidad de Guanajuato, León, Guanajuato, México
| | | | - Fernando A. Barrios
- Universidad Nacional Autónoma de México, Instituto de Neurobiología, Querétaro, México
- * E-mail:
| |
Collapse
|
46
|
Zinchenko O, Yaple ZA, Arsalidou M. Brain Responses to Dynamic Facial Expressions: A Normative Meta-Analysis. Front Hum Neurosci 2018; 12:227. [PMID: 29922137 PMCID: PMC5996092 DOI: 10.3389/fnhum.2018.00227] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2018] [Accepted: 05/16/2018] [Indexed: 01/08/2023] Open
Abstract
Identifying facial expressions is crucial for social interactions. Functional neuroimaging studies show that a set of brain areas, such as the fusiform gyrus and amygdala, become active when viewing emotional facial expressions. The majority of functional magnetic resonance imaging (fMRI) studies investigating face perception typically employ static images of faces. However, studies that use dynamic facial expressions (e.g., videos) are accumulating and suggest that a dynamic presentation may be more sensitive and ecologically valid for investigating faces. By using quantitative fMRI meta-analysis the present study examined concordance of brain regions associated with viewing dynamic facial expressions. We analyzed data from 216 participants that participated in 14 studies, which reported coordinates for 28 experiments. Our analysis revealed bilateral fusiform and middle temporal gyri, left amygdala, left declive of the cerebellum and the right inferior frontal gyrus. These regions are discussed in terms of their relation to models of face processing.
Collapse
Affiliation(s)
- Oksana Zinchenko
- Centre for Cognition and Decision Making, Institute for Cognitive Neuroscience, National Research University Higher School of Economics, Moscow, Russia
| | - Zachary A Yaple
- Centre for Cognition and Decision Making, Institute for Cognitive Neuroscience, National Research University Higher School of Economics, Moscow, Russia.,Department of Psychology, National University of Singapore, Singapore, Singapore
| | - Marie Arsalidou
- Department of Psychology, National Research University Higher School of Economics, Moscow, Russia.,Department of Psychology, York University, Toronto, ON, Canada
| |
Collapse
|
47
|
Sarkheil P, Kilian-Hütten N, Mickartz K, Vornholt T, Mathiak K. Variation of temporal order reveals deficits in categorisation of facial expressions in patients afflicted with depression. Cogn Neuropsychiatry 2018; 23:154-164. [PMID: 29502459 DOI: 10.1080/13546805.2018.1444596] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
Abstract
INTRODUCTION It is well established that depressive disorders are associated with abnormalities in the processing of affective information. However, type of stimuli, perceptual complexity and cognitive demand are important factors in evaluating these findings. In particular, processing mechanisms of perceptual boundaries in ecologically valid face stimuli are largely unknown in depression. METHODS In the current study, intensity-ordered frame sequences provided a dynamic visualisation of happy or sad facial expressions fading from or to neutral expressions. Patients (n = 20) with major depression (MD) disorder and controls (n = 20) indicated their perceptual boundaries between neutral and emotional face depending on direction and emotion. The averaged time of the perceptual boundary entered a group × condition ANOVA and regression analysis. RESULTS MD group did not systematically shift perceptual boundaries in the dynamic emotional faces but yielded altered statistics in information processing. The Gaussian distribution of boundary judgements was disturbed in depression, increasing goodness-of-fit errors in disappearing emotions. Goodness-of-fit correlated with depression symptom score (Beck Depression Inventory-II (BDI-II)) in the MD group during the disappearing sad (r(18) = 46, p = 0.04) and happy (r(18) = 51, p = 0.02) conditions. CONCLUSION We evaluated detection of appearing and disappearing emotions in dynamic faces. A deviant distribution of categorisation responses emerged in the MD group, which was not emotion-specific. Such a perceptional uncertainty can impede individuals' functioning in interpersonal interaction.
Collapse
Affiliation(s)
- Pegah Sarkheil
- a Department of Psychiatry, Psychotherapy, and Psychosomatics, Medical Faculty , RWTH Aachen University , Aachen , Germany.,c JARA-Translational Brain Medicine , RWTH Aachen , Aachen , Germany
| | - Niclas Kilian-Hütten
- b Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience , Maastricht University , Maastricht , The Netherlands
| | - Kristina Mickartz
- a Department of Psychiatry, Psychotherapy, and Psychosomatics, Medical Faculty , RWTH Aachen University , Aachen , Germany.,b Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience , Maastricht University , Maastricht , The Netherlands
| | - Thomas Vornholt
- a Department of Psychiatry, Psychotherapy, and Psychosomatics, Medical Faculty , RWTH Aachen University , Aachen , Germany.,b Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience , Maastricht University , Maastricht , The Netherlands
| | - Klaus Mathiak
- a Department of Psychiatry, Psychotherapy, and Psychosomatics, Medical Faculty , RWTH Aachen University , Aachen , Germany.,c JARA-Translational Brain Medicine , RWTH Aachen , Aachen , Germany
| |
Collapse
|
48
|
Calvo MG, Krumhuber EG, Fernández-Martín A. Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions. Q J Exp Psychol (Hove) 2018; 72:729-741. [PMID: 29471708 DOI: 10.1177/1747021818763747] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Abstract
A happy facial expression makes a person look (more) trustworthy. Do perceptions of happiness and trustworthiness rely on the same face regions and visual attention processes? In an eye-tracking study, eye movements and fixations were recorded while participants judged the un/happiness or the un/trustworthiness of dynamic facial expressions in which the eyes and/or the mouth unfolded from neutral to happy or vice versa. A smiling mouth and happy eyes enhanced perceived happiness and trustworthiness similarly, with a greater contribution of the smile relative to the eyes. This comparable judgement output for happiness and trustworthiness was reached through shared as well as distinct attentional mechanisms: (a) entry times and (b) initial fixation thresholds for each face region were equivalent for both judgements, thereby revealing the same attentional orienting in happiness and trustworthiness processing. However, (c) greater and (d) longer fixation density for the mouth region in the happiness task, and for the eye region in the trustworthiness task, demonstrated different selective attentional engagement. Relatedly, (e) mean fixation duration across face regions was longer in the trustworthiness task, thus showing increased attentional intensity or processing effort.
Collapse
Affiliation(s)
- Manuel G Calvo
- 1 Department of Cognitive Psychology, Universidad de La Laguna, Tenerife, Spain
| | | | - Andrés Fernández-Martín
- 3 Universidad Internacional de La Rioja, Logroño, Spain and Instituto Universitario de Neurociencia (IUNE), Universidad de La Laguna, Tenerife, Spain
| |
Collapse
|
49
|
Liang Y, Liu B, Li X, Wang P. Multivariate Pattern Classification of Facial Expressions Based on Large-Scale Functional Connectivity. Front Hum Neurosci 2018; 12:94. [PMID: 29615882 PMCID: PMC5868121 DOI: 10.3389/fnhum.2018.00094] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2017] [Accepted: 02/27/2018] [Indexed: 01/15/2023] Open
Abstract
It is an important question how human beings achieve efficient recognition of others’ facial expressions in cognitive neuroscience, and it has been identified that specific cortical regions show preferential activation to facial expressions in previous studies. However, the potential contributions of the connectivity patterns in the processing of facial expressions remained unclear. The present functional magnetic resonance imaging (fMRI) study explored whether facial expressions could be decoded from the functional connectivity (FC) patterns using multivariate pattern analysis combined with machine learning algorithms (fcMVPA). We employed a block design experiment and collected neural activities while participants viewed facial expressions of six basic emotions (anger, disgust, fear, joy, sadness, and surprise). Both static and dynamic expression stimuli were included in our study. A behavioral experiment after scanning confirmed the validity of the facial stimuli presented during the fMRI experiment with classification accuracies and emotional intensities. We obtained whole-brain FC patterns for each facial expression and found that both static and dynamic facial expressions could be successfully decoded from the FC patterns. Moreover, we identified the expression-discriminative networks for the static and dynamic facial expressions, which span beyond the conventional face-selective areas. Overall, these results reveal that large-scale FC patterns may also contain rich expression information to accurately decode facial expressions, suggesting a novel mechanism, which includes general interactions between distributed brain regions, and that contributes to the human facial expression recognition.
Collapse
Affiliation(s)
- Yin Liang
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Baolin Liu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China.,State Key Laboratory of Intelligent Technology and Systems, National Laboratory for Information Science and Technology, Tsinghua University, Beijing, China
| | - Xianglin Li
- Medical Imaging Research Institute, Binzhou Medical University, Yantai, China
| | - Peiyuan Wang
- Department of Radiology, Yantai Affiliated Hospital of Binzhou Medical University, Yantai, China
| |
Collapse
|
50
|
Dobs K, Schultz J, Bülthoff I, Gardner JL. Task-dependent enhancement of facial expression and identity representations in human cortex. Neuroimage 2018; 172:689-702. [PMID: 29432802 DOI: 10.1016/j.neuroimage.2018.02.013] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2017] [Revised: 02/02/2018] [Accepted: 02/06/2018] [Indexed: 11/24/2022] Open
Abstract
What cortical mechanisms allow humans to easily discern the expression or identity of a face? Subjects detected changes in expression or identity of a stream of dynamic faces while we measured BOLD responses from topographically and functionally defined areas throughout the visual hierarchy. Responses in dorsal areas increased during the expression task, whereas responses in ventral areas increased during the identity task, consistent with previous studies. Similar to ventral areas, early visual areas showed increased activity during the identity task. If visual responses are weighted by perceptual mechanisms according to their magnitude, these increased responses would lead to improved attentional selection of the task-appropriate facial aspect. Alternatively, increased responses could be a signature of a sensitivity enhancement mechanism that improves representations of the attended facial aspect. Consistent with the latter sensitivity enhancement mechanism, attending to expression led to enhanced decoding of exemplars of expression both in early visual and dorsal areas relative to attending identity. Similarly, decoding identity exemplars when attending to identity was improved in dorsal and ventral areas. We conclude that attending to expression or identity of dynamic faces is associated with increased selectivity in representations consistent with sensitivity enhancement.
Collapse
Affiliation(s)
- Katharina Dobs
- Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Max-Planck-Ring 8, 72076 Tübingen, Germany; Laboratory for Human Systems Neuroscience, RIKEN Brain Science Institute, 2-1 Hirosawa, Wako, Saitama 351-0198, Japan; Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar Street, Cambridge, MA 02139, USA.
| | - Johannes Schultz
- Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Max-Planck-Ring 8, 72076 Tübingen, Germany; Division of Medical Psychology and Department of Psychiatry, University of Bonn, Sigmund Freud Str. 25, 53105 Bonn, Germany
| | - Isabelle Bülthoff
- Department Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Max-Planck-Ring 8, 72076 Tübingen, Germany
| | - Justin L Gardner
- Laboratory for Human Systems Neuroscience, RIKEN Brain Science Institute, 2-1 Hirosawa, Wako, Saitama 351-0198, Japan; Department of Psychology, Stanford University, 450 Serra Mall, Stanford, CA 94305, USA
| |
Collapse
|