1
|
Park C, Kim J. Taste the music: Modality-general representation of affective states derived from auditory and gustatory stimuli. Cognition 2024; 249:105830. [PMID: 38810426 DOI: 10.1016/j.cognition.2024.105830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 05/12/2024] [Accepted: 05/21/2024] [Indexed: 05/31/2024]
Abstract
Prior studies have extensively examined modality-general representation of affect across various sensory modalities, particularly focusing on auditory and visual stimuli. However, little research has explored the modality-general representation of affect between gustatory and other sensory modalities. This study aimed to investigate whether the affective responses induced by tastes and musical pieces could be predicted within and across modalities. For each modality, eight stimuli were chosen based on four basic taste conditions (sweet, bitter, sour, and salty). Participants rated their responses to each stimulus using both taste and emotion scales. The multivariate analyses including multidimensional scaling and classification analysis were performed. The findings revealed that auditory and gustatory stimuli in the sweet category were associated with positive valence, whereas those from the other taste categories were linked to negative valence. Additionally, auditory and gustatory stimuli in sour taste category were linked to high arousal, whereas stimuli in bitter taste category were associated with low arousal. This study revealed the potential mapping of gustatory and auditory stimuli onto core affect space in everyday experiences. Moreover, it demonstrated that emotions evoked by taste and music could be predicted across modalities, supporting modality-general representation of affect.
Collapse
Affiliation(s)
- Chaery Park
- Jeonbuk National University, Jeonsu-si, South Korea, 567 Baekje-daero, Deokjin-gu, Jeonju-si, Jeollabuk-do 54896, Republic of Korea
| | - Jongwan Kim
- Jeonbuk National University, Jeonsu-si, South Korea, 567 Baekje-daero, Deokjin-gu, Jeonju-si, Jeollabuk-do 54896, Republic of Korea.
| |
Collapse
|
2
|
Lee KM, Satpute AB. More than labels: neural representations of emotion words are widely distributed across the brain. Soc Cogn Affect Neurosci 2024; 19:nsae043. [PMID: 38903026 PMCID: PMC11259136 DOI: 10.1093/scan/nsae043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 03/15/2024] [Accepted: 06/20/2024] [Indexed: 06/22/2024] Open
Abstract
Although emotion words such as "anger," "disgust," "happiness," or "pride" are often thought of as mere labels, increasing evidence points to language as being important for emotion perception and experience. Emotion words may be particularly important for facilitating access to the emotion concepts. Indeed, deficits in semantic processing or impaired access to emotion words interfere with emotion perception. Yet, it is unclear what these behavioral findings mean for affective neuroscience. Thus, we examined the brain areas that support processing of emotion words using representational similarity analysis of functional magnetic resonance imaging data (N = 25). In the task, participants saw 10 emotion words (e.g. "anger," "happiness") while in the scanner. Participants rated each word based on its valence on a continuous scale ranging from 0 (Pleasant/Good) to 1 (Unpleasant/Bad) scale to ensure they were processing the words. Our results revealed that a diverse range of brain areas including prefrontal, midline cortical, and sensorimotor regions contained information about emotion words. Notably, our results overlapped with many regions implicated in decoding emotion experience by prior studies. Our results raise questions about what processes are being supported by these regions during emotion experience.
Collapse
Affiliation(s)
- Kent M Lee
- Department of Psychology, Northeastern University, 125 Nightingale Hall, Boston, MA 02115, USA
| | - Ajay B Satpute
- Department of Psychology, Northeastern University, 125 Nightingale Hall, Boston, MA 02115, USA
| |
Collapse
|
3
|
Lee SA, Lee JJ, Han J, Choi M, Wager TD, Woo CW. Brain representations of affective valence and intensity in sustained pleasure and pain. Proc Natl Acad Sci U S A 2024; 121:e2310433121. [PMID: 38857402 PMCID: PMC11194486 DOI: 10.1073/pnas.2310433121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 04/18/2024] [Indexed: 06/12/2024] Open
Abstract
Pleasure and pain are two fundamental, intertwined aspects of human emotions. Pleasurable sensations can reduce subjective feelings of pain and vice versa, and we often perceive the termination of pain as pleasant and the absence of pleasure as unpleasant. This implies the existence of brain systems that integrate them into modality-general representations of affective experiences. Here, we examined representations of affective valence and intensity in an functional MRI (fMRI) study (n = 58) of sustained pleasure and pain. We found that the distinct subpopulations of voxels within the ventromedial and lateral prefrontal cortices, the orbitofrontal cortex, the anterior insula, and the amygdala were involved in decoding affective valence versus intensity. Affective valence and intensity predictive models showed significant decoding performance in an independent test dataset (n = 62). These models were differentially connected to distinct large-scale brain networks-the intensity model to the ventral attention network and the valence model to the limbic and default mode networks. Overall, this study identified the brain representations of affective valence and intensity across pleasure and pain, promoting a systems-level understanding of human affective experiences.
Collapse
Affiliation(s)
- Soo Ahn Lee
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, Republic of Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, Republic of Korea
- Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon16419, Republic of Korea
| | - Jae-Joong Lee
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, Republic of Korea
| | - Jisoo Han
- Korea Brain Research Institute, Daegu41062, Republic of Korea
| | - Myunghwan Choi
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, Republic of Korea
- School of Biological Sciences, Seoul National University, Seoul08826, Republic of Korea
| | - Tor D. Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH03755
| | - Choong-Wan Woo
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, Republic of Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, Republic of Korea
- Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon16419, Republic of Korea
- Life-inspired Neural Network for Prediction and Optimization Research Group, Suwon16419, Republic of Korea
| |
Collapse
|
4
|
Lee S, Kim J. Testing the bipolar assumption of Singer-Loomis Type Deployment Inventory for Korean adults using classification and multidimensional scaling. Front Psychol 2024; 14:1249185. [PMID: 38356992 PMCID: PMC10864660 DOI: 10.3389/fpsyg.2023.1249185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 12/26/2023] [Indexed: 02/16/2024] Open
Abstract
In this study, we explored whether the Korean version of Singer Loomis Type Deployment Inventory II (K-SLTDI) captures the opposing tendencies of Jung's theory of psychological type. The types are Extroverted Sensing, Extroverted Intuition, Extroverted Feeling, Extroverted Thinking, Introverted Sensing, Introverted Intuition, Introverted Feeling, and Introverted Thinking. A nationwide online survey was conducted in South Korea. We performed multidimensional scaling and classification analyses based on 521 Korean adult profiles with eight psychological types to test the bipolarity assumption. The results showed that the Procrustes-rotated four-dimensional space successfully represented four types of opposing tendencies. Moreover, the bipolarity assumption in the four dimensions of Jungian typology was tested and compared between lower and higher psychological distress populations via cluster analysis. Lastly, we explored patterns of responses in lower and higher psychological distress populations using intersubject correlation. Both similarity analyses and classification results consistently support the theoretical considerations on the conceptualization of Jung's type in independent order that the types could be derived without bipolar assumption as Singer and Loomis expected in their Type Development Inventory. Limitations in our study include the sample being randomly selected internet users during the COVID-19 pandemic, despite excellence in the use of the internet in the general Korean population.
Collapse
Affiliation(s)
| | - Jongwan Kim
- Psychology Department, Jeonbuk National University, Jeonju, Republic of Korea
| |
Collapse
|
5
|
Grogans SE, Bliss-Moreau E, Buss KA, Clark LA, Fox AS, Keltner D, Cowen AS, Kim JJ, Kragel PA, MacLeod C, Mobbs D, Naragon-Gainey K, Fullana MA, Shackman AJ. The nature and neurobiology of fear and anxiety: State of the science and opportunities for accelerating discovery. Neurosci Biobehav Rev 2023; 151:105237. [PMID: 37209932 PMCID: PMC10330657 DOI: 10.1016/j.neubiorev.2023.105237] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Revised: 05/11/2023] [Accepted: 05/13/2023] [Indexed: 05/22/2023]
Abstract
Fear and anxiety play a central role in mammalian life, and there is considerable interest in clarifying their nature, identifying their biological underpinnings, and determining their consequences for health and disease. Here we provide a roundtable discussion on the nature and biological bases of fear- and anxiety-related states, traits, and disorders. The discussants include scientists familiar with a wide variety of populations and a broad spectrum of techniques. The goal of the roundtable was to take stock of the state of the science and provide a roadmap to the next generation of fear and anxiety research. Much of the discussion centered on the key challenges facing the field, the most fruitful avenues for future research, and emerging opportunities for accelerating discovery, with implications for scientists, funders, and other stakeholders. Understanding fear and anxiety is a matter of practical importance. Anxiety disorders are a leading burden on public health and existing treatments are far from curative, underscoring the urgency of developing a deeper understanding of the factors governing threat-related emotions.
Collapse
Affiliation(s)
- Shannon E Grogans
- Department of Psychology, University of Maryland, College Park, MD 20742, USA
| | - Eliza Bliss-Moreau
- Department of Psychology, University of California, Davis, CA 95616, USA; California National Primate Research Center, University of California, Davis, CA 95616, USA
| | - Kristin A Buss
- Department of Psychology, The Pennsylvania State University, University Park, PA 16802 USA
| | - Lee Anna Clark
- Department of Psychology, University of Notre Dame, Notre Dame, IN 46556, USA
| | - Andrew S Fox
- Department of Psychology, University of California, Davis, CA 95616, USA; California National Primate Research Center, University of California, Davis, CA 95616, USA
| | - Dacher Keltner
- Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA
| | | | - Jeansok J Kim
- Department of Psychology, University of Washington, Seattle, WA 98195, USA
| | - Philip A Kragel
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - Colin MacLeod
- Centre for the Advancement of Research on Emotion, School of Psychological Science, The University of Western Australia, Perth, WA 6009, Australia
| | - Dean Mobbs
- Department of Humanities and Social Sciences, California Institute of Technology, Pasadena, California 91125, USA; Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA 91125, USA
| | - Kristin Naragon-Gainey
- School of Psychological Science, University of Western Australia, Perth, WA 6009, Australia
| | - Miquel A Fullana
- Adult Psychiatry and Psychology Department, Institute of Neurosciences, Hospital Clinic, Barcelona, Spain; Imaging of Mood, and Anxiety-Related Disorders Group, Institut d'Investigacions Biomèdiques August Pi i Sunyer, CIBERSAM, University of Barcelona, Barcelona, Spain
| | - Alexander J Shackman
- Department of Psychology, University of Maryland, College Park, MD 20742, USA; Neuroscience and Cognitive Science Program, University of Maryland, College Park, MD 20742, USA; Maryland Neuroimaging Center, University of Maryland, College Park, MD 20742, USA.
| |
Collapse
|
6
|
Common and stimulus-type-specific brain representations of negative affect. Nat Neurosci 2022; 25:760-770. [DOI: 10.1038/s41593-022-01082-w] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2021] [Accepted: 04/25/2022] [Indexed: 01/16/2023]
|
7
|
Kim MJ, Knodt AR, Hariri AR. Meta-Analytic Activation Maps Can Help Identify Affective Processes Captured by Contrast-Based Task fMRI: The Case of Threat-Related Facial Expressions. Soc Cogn Affect Neurosci 2022; 17:777-787. [PMID: 35137241 PMCID: PMC9433847 DOI: 10.1093/scan/nsac010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Revised: 10/30/2021] [Accepted: 02/07/2022] [Indexed: 12/02/2022] Open
Abstract
Meta-analysis of functional magnetic resonance imaging (fMRI) data is an effective method for capturing the distributed patterns of brain activity supporting discrete cognitive and affective processes. One opportunity presented by the resulting meta-analysis maps (MAMs) is as a reference for better understanding the nature of individual contrast maps (ICMs) derived from specific task fMRI data. Here, we compared MAMs from 148 neuroimaging studies representing emotion categories of fear, anger, disgust, happiness and sadness with ICMs from fearful > neutral and angry > neutral faces from an independent dataset of task fMRI (n = 1263). Analyses revealed that both fear and anger ICMs exhibited the greatest pattern similarity to fear MAMs. As the number of voxels included for the computation of pattern similarity became more selective, the specificity of MAM–ICM correspondence decreased. Notably, amygdala activity long considered critical for processing threat-related facial expressions was neither sufficient nor necessary for detecting MAM–ICM pattern similarity effects. Our analyses suggest that both fearful and angry facial expressions are best captured by distributed patterns of brain activity, a putative neural correlate of threat. More generally, our analyses demonstrate how MAMs can be leveraged to better understand affective processes captured by ICMs in task fMRI data.
Collapse
Affiliation(s)
- M Justin Kim
- Correspondence should be addressed to M. Justin Kim, Department of Psychology, Sungkyunkwan University, Seoul 03063, South Korea. E-mail:
| | - Annchen R Knodt
- Laboratory of NeuroGenetics, Department of Psychology and Neuroscience, Duke University, Durham, NC 27708 USA
| | - Ahmad R Hariri
- Laboratory of NeuroGenetics, Department of Psychology and Neuroscience, Duke University, Durham, NC 27708 USA
| |
Collapse
|
8
|
Lettieri G, Handjaras G, Setti F, Cappello EM, Bruno V, Diano M, Leo A, Ricciardi E, Pietrini P, Cecchetti L. Default and Control Networks Connectivity Dynamics Track the Stream of Affect at Multiple Timescales. Soc Cogn Affect Neurosci 2021; 17:461-469. [PMID: 34673987 PMCID: PMC9071410 DOI: 10.1093/scan/nsab112] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2021] [Revised: 09/24/2021] [Accepted: 10/21/2021] [Indexed: 11/14/2022] Open
Abstract
In everyday life the stream of affect results from the interaction between past experiences, expectations, and the unfolding of events. How the brain represents the relationship between time and affect has been hardly explored, as it requires modeling the complexity of everyday life in the laboratory setting. Movies condense into hours a multitude of emotional responses, synchronized across subjects and characterized by temporal dynamics alike real-world experiences. Here, we use time-varying intersubject brain synchronization and real-time behavioral reports to test whether connectivity dynamics track changes in affect during movie watching. Results show that polarity and intensity of experiences relate to connectivity of the default mode and control networks and converge in the right temporo-parietal cortex. We validate these results in two experiments including four independent samples, two movies, and alternative analysis workflows. Lastly, we reveal chronotopic connectivity maps within temporo-parietal and prefrontal cortex, where adjacent areas preferentially encode affect at specific timescales.
Collapse
Affiliation(s)
- Giada Lettieri
- Social and Affective Neuroscience Group, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Giacomo Handjaras
- Social and Affective Neuroscience Group, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Francesca Setti
- Molecular Mind Laboratory, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Elisa Morgana Cappello
- Social and Affective Neuroscience Group, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Valentina Bruno
- MANIBUS Lab, Psychology Department, University of Turin, Turin, Italy
| | - Matteo Diano
- Department of Psychology, University of Turin, Turin, Italy
| | - Andrea Leo
- Department of Translational Research and Advanced Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy
| | - Emiliano Ricciardi
- Molecular Mind Laboratory, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Pietro Pietrini
- Molecular Mind Laboratory, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Luca Cecchetti
- Social and Affective Neuroscience Group, IMT School for Advanced Studies Lucca, Lucca, Italy
| |
Collapse
|
9
|
Sievers B, Parkinson C, Kohler PJ, Hughes JM, Fogelson SV, Wheatley T. Visual and auditory brain areas share a representational structure that supports emotion perception. Curr Biol 2021; 31:5192-5203.e4. [PMID: 34644547 DOI: 10.1016/j.cub.2021.09.043] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Revised: 07/07/2021] [Accepted: 09/16/2021] [Indexed: 11/18/2022]
Abstract
Emotionally expressive music and dance occur together across the world. This may be because features shared across the senses are represented the same way even in different sensory brain areas, putting music and movement in directly comparable terms. These shared representations may arise from a general need to identify environmentally relevant combinations of sensory features, particularly those that communicate emotion. To test the hypothesis that visual and auditory brain areas share a representational structure, we created music and animation stimuli with crossmodally matched features expressing a range of emotions. Participants confirmed that each emotion corresponded to a set of features shared across music and movement. A subset of participants viewed both music and animation during brain scanning, revealing that representations in auditory and visual brain areas were similar to one another. This shared representation captured not only simple stimulus features but also combinations of features associated with emotion judgments. The posterior superior temporal cortex represented both music and movement using this same structure, suggesting supramodal abstraction of sensory content. Further exploratory analysis revealed that early visual cortex used this shared representational structure even when stimuli were presented auditorily. We propose that crossmodally shared representations support mutually reinforcing dynamics across auditory and visual brain areas, facilitating crossmodal comparison. These shared representations may help explain why emotions are so readily perceived and why some dynamic emotional expressions can generalize across cultural contexts.
Collapse
Affiliation(s)
- Beau Sievers
- Department of Psychology, Harvard University, Cambridge, MA 02138, USA; Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755, USA.
| | - Carolyn Parkinson
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA; Brain Research Institute, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Peter J Kohler
- Department of Psychology, York University, Toronto, ON, Canada; Centre for Vision Research, York University, Toronto, ON, Canada
| | | | | | - Thalia Wheatley
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755, USA; Santa Fe Institute, Santa Fe, NM 87501, USA.
| |
Collapse
|
10
|
Lee KM, Ferreira-Santos F, Satpute AB. Predictive processing models and affective neuroscience. Neurosci Biobehav Rev 2021; 131:211-228. [PMID: 34517035 DOI: 10.1016/j.neubiorev.2021.09.009] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 02/10/2021] [Accepted: 09/07/2021] [Indexed: 01/17/2023]
Abstract
The neural bases of affective experience remain elusive. Early neuroscience models of affect searched for specific brain regions that uniquely carried out the computations that underlie dimensions of valence and arousal. However, a growing body of work has failed to identify these circuits. Research turned to multivariate analyses, but these strategies, too, have made limited progress. Predictive processing models offer exciting new directions to address this problem. Here, we use predictive processing models as a lens to critique prevailing functional neuroimaging research practices in affective neuroscience. Our review highlights how much work relies on rigid assumptions that are inconsistent with a predictive processing approach. We outline the central aspects of a predictive processing model and draw out their implications for research in affective and cognitive neuroscience. Predictive models motivate a reformulation of "reverse inference" in cognitive neuroscience, and placing a greater emphasis on external validity in experimental design.
Collapse
Affiliation(s)
- Kent M Lee
- Northeastern University, 360 Huntington Ave, 125 NI, Boston, MA 02118, USA.
| | - Fernando Ferreira-Santos
- Laboratory of Neuropsychophysiology, Faculty of Psychology and Education Sciences, University of Porto, Portugal
| | - Ajay B Satpute
- Northeastern University, 360 Huntington Ave, 125 NI, Boston, MA 02118, USA
| |
Collapse
|
11
|
Abstract
Initial evaluation structures (IESs) currently proposed as the earliest detectors of affective stimuli (e.g., amygdala, orbitofrontal cortex, or insula) are high-order structures (a) whose response latency cannot account for the first visual cortex emotion-related response (~80 ms), and (b) lack the necessary infrastructure to locally analyze the visual features that define emotional stimuli. Several thalamic structures accomplish both criteria. The lateral geniculate nucleus (LGN), a first-order thalamic nucleus that actively processes visual information, with the complement of the thalamic reticular nucleus (TRN) are proposed as core IESs. This LGN–TRN tandem could be supported by the pulvinar, a second-order thalamic structure, and by other extrathalamic nuclei. The visual thalamus, scarcely explored in affective neurosciences, seems crucial in early emotional evaluation.
Collapse
Affiliation(s)
- Luis Carretié
- Facultad de Psicología, Universidad Autónoma de Madrid, Spain
| | | | | |
Collapse
|
12
|
Xu J, Dong H, Li N, Wang Z, Guo F, Wei J, Dang J. Weighted RSA: An Improved Framework on the Perception of Audio-visual Affective Speech in Left Insula and Superior Temporal Gyrus. Neuroscience 2021; 469:46-58. [PMID: 34119576 DOI: 10.1016/j.neuroscience.2021.06.002] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2021] [Revised: 05/16/2021] [Accepted: 06/02/2021] [Indexed: 12/24/2022]
Abstract
Being able to accurately perceive the emotion expressed by the facial or verbal expression from others is critical to successful social interaction. However, only few studies examined the multimodal interactions on speech emotion, and there is no consistence in studies on the speech emotion perception. It remains unclear, how the speech emotion of different valence is perceived on the multimodal stimuli by our human brain. In this paper, we conducted a functional magnetic resonance imaging (fMRI) study with an event-related design, using dynamic facial expressions and emotional speech stimuli to express different emotions, in order to explore the perception mechanism of speech emotion in audio-visual modality. The representational similarity analysis (RSA), whole-brain searchlight analysis, and conjunction analysis of emotion were used to interpret the representation of speech emotion in different aspects. Significantly, a weighted RSA approach was creatively proposed to evaluate the contribution of each candidate model to the best fitted model and provided a supplement to RSA. The results of weighted RSA indicated that the fitted models were superior to all candidate models and the weights could be used to explain the representation of ROIs. The bilateral amygdala has been shown to be associated with the processing of both positive and negative emotions except neutral emotion. It is indicated that the left posterior insula and the left anterior superior temporal gyrus (STG) play important roles in the perception of multimodal speech emotion.
Collapse
Affiliation(s)
- Junhai Xu
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Haibin Dong
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China; State Grid Tianjin Electric Power Company, China
| | - Na Li
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Zeyu Wang
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Fei Guo
- School of Computer Science and Engineering, Central South University, Changsha 410083, China.
| | - Jianguo Wei
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China.
| | - Jianwu Dang
- College of Intelligence and Computing, Tianjin Key Lab of Cognitive Computing and Application, Tianjin University, Tianjin, China; School of Information Science, Japan Advanced Institute of Science and Technology, Japan
| |
Collapse
|
13
|
Koelsch S, Cheung VKM, Jentschke S, Haynes JD. Neocortical substrates of feelings evoked with music in the ACC, insula, and somatosensory cortex. Sci Rep 2021; 11:10119. [PMID: 33980876 PMCID: PMC8115666 DOI: 10.1038/s41598-021-89405-y] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Accepted: 04/21/2021] [Indexed: 12/01/2022] Open
Abstract
Neurobiological models of emotion focus traditionally on limbic/paralimbic regions as neural substrates of emotion generation, and insular cortex (in conjunction with isocortical anterior cingulate cortex, ACC) as the neural substrate of feelings. An emerging view, however, highlights the importance of isocortical regions beyond insula and ACC for the subjective feeling of emotions. We used music to evoke feelings of joy and fear, and multivariate pattern analysis (MVPA) to decode representations of feeling states in functional magnetic resonance (fMRI) data of n = 24 participants. Most of the brain regions providing information about feeling representations were neocortical regions. These included, in addition to granular insula and cingulate cortex, primary and secondary somatosensory cortex, premotor cortex, frontal operculum, and auditory cortex. The multivoxel activity patterns corresponding to feeling representations emerged within a few seconds, gained in strength with increasing stimulus duration, and replicated results of a hypothesis-generating decoding analysis from an independent experiment. Our results indicate that several neocortical regions (including insula, cingulate, somatosensory and premotor cortices) are important for the generation and modulation of feeling states. We propose that secondary somatosensory cortex, which covers the parietal operculum and encroaches on the posterior insula, is of particular importance for the encoding of emotion percepts, i.e., preverbal representations of subjective feeling.
Collapse
Affiliation(s)
- Stefan Koelsch
- Department of Biological and Medical Psychology, University of Bergen, Bergen, Norway. .,Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| | - Vincent K M Cheung
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Institute of Information Science, Academia Sinica, Taipei, Taiwan
| | | | - John-Dylan Haynes
- Berlin Center for Advanced Neuroimaging, Charité - Universitätsmedizin Berlin, Berlin, Germany
| |
Collapse
|
14
|
Putkinen V, Nazari-Farsani S, Seppälä K, Karjalainen T, Sun L, Karlsson HK, Hudson M, Heikkilä TT, Hirvonen J, Nummenmaa L. Decoding Music-Evoked Emotions in the Auditory and Motor Cortex. Cereb Cortex 2021; 31:2549-2560. [PMID: 33367590 DOI: 10.1093/cercor/bhaa373] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2020] [Revised: 10/16/2020] [Accepted: 11/06/2020] [Indexed: 11/14/2022] Open
Abstract
Music can induce strong subjective experience of emotions, but it is debated whether these responses engage the same neural circuits as emotions elicited by biologically significant events. We examined the functional neural basis of music-induced emotions in a large sample (n = 102) of subjects who listened to emotionally engaging (happy, sad, fearful, and tender) pieces of instrumental music while their hemodynamic brain activity was measured with functional magnetic resonance imaging (fMRI). Ratings of the four categorical emotions and liking were used to predict hemodynamic responses in general linear model (GLM) analysis of the fMRI data. Multivariate pattern analysis (MVPA) was used to reveal discrete neural signatures of the four categories of music-induced emotions. To map neural circuits governing non-musical emotions, the subjects were scanned while viewing short emotionally evocative film clips. The GLM revealed that most emotions were associated with activity in the auditory, somatosensory, and motor cortices, cingulate gyrus, insula, and precuneus. Fear and liking also engaged the amygdala. In contrast, the film clips strongly activated limbic and cortical regions implicated in emotional processing. MVPA revealed that activity in the auditory cortex and primary motor cortices reliably discriminated the emotion categories. Our results indicate that different music-induced basic emotions have distinct representations in regions supporting auditory processing, motor control, and interoception but do not strongly rely on limbic and medial prefrontal regions critical for emotions with survival value.
Collapse
Affiliation(s)
- Vesa Putkinen
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Sanaz Nazari-Farsani
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Kerttu Seppälä
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Tomi Karjalainen
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Lihua Sun
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Henry K Karlsson
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland
| | - Matthew Hudson
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland.,National College of Ireland, D01 K6W2, Dublin, Ireland
| | - Timo T Heikkilä
- Department of Psychology, University of Turku, FI-20014, Turku, Finland
| | - Jussi Hirvonen
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland.,Department of Radiology, Turku University Hospital, 20520, Turku, Finland
| | - Lauri Nummenmaa
- Turku PET Centre, and Turku University Hospital, University of Turku, 20520, Turku, Finland.,Department of Psychology, University of Turku, FI-20014, Turku, Finland
| |
Collapse
|
15
|
Modality-general and modality-specific audiovisual valence processing. Cortex 2021; 138:127-137. [PMID: 33684626 DOI: 10.1016/j.cortex.2021.01.022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2020] [Revised: 11/08/2020] [Accepted: 01/20/2021] [Indexed: 11/23/2022]
Abstract
A fundamental question in affective neuroscience is whether there is a common hedonic system for valence processing independent of modality, or there are distinct neural systems for different modalities. To address this question, we used both region of interest and whole-brain representational similarity analyses on functional magnetic resonance imaging data to identify modality-general and modality-specific brain areas involved in valence processing across visual and auditory modalities. First, region of interest analyses showed that the superior temporal cortex was associated with both modality-general and auditory-specific models, while the primary visual cortex was associated with the visual-specific model. Second, the whole-brain searchlight analyses also identified both modality-general and modality-specific representations. The modality-general regions included the superior temporal, medial superior frontal, inferior frontal, precuneus, precentral, postcentral, supramarginal, paracentral lobule and middle cingulate cortices. The modality-specific regions included both perceptual cortices and higher-order brain areas. The valence representations derived from individualized behavioral valence ratings were consistent with these results. Together, these findings suggest both modality-general and modality-specific representations of valence.
Collapse
|
16
|
Hama T, Koeda M, Ikeda Y, Tateno A, Kawara T, Suzuki H, Okubo Y. Bupropion increases cerebral activation in auditory affective processing: A randomized controlled fMRI study. Neurosci Lett 2021; 749:135716. [PMID: 33592303 DOI: 10.1016/j.neulet.2021.135716] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2020] [Revised: 02/01/2021] [Accepted: 02/04/2021] [Indexed: 10/22/2022]
Abstract
INTRODUCTION Bupropion is an antidepressant with less possibility to give rise to emotional blunting as side effect, and it also acts on improving negative self-recognition in a depressive state. Previous neuroimaging studies indicated a change in brain function by facial expression as an effect of antidepressants. As well as facial expression, vocal affective processing is essential for accurately recognizing another's feelings, but to our knowledge, no study has investigated whether bupropion affects the cerebral function of recognition of auditory affective processing. In this study, we aimed to investigate the acute effect of bupropion on cerebral response to vocal affective processing. METHODS Sixteen healthy volunteers (male = 8) participated in this study. With a randomized placebo-controlled within-subject trial, two series of fMRI scans, using either placebo or bupropion (150 mg), were examined. An auditory emotional valence judgement task was performed during fMRI scanning. The acute effects of bupropion on cerebral activation in the emotional circuit and behavioral performance during emotional processing were analyzed. RESULTS Compared with placebo, bupropion caused a significantly greater activation of emotional voices in the left insula and right superior temporal gyrus, whereas the amygdala was not activated. By bupropion, a significantly greater activation of the positive emotional circuit was observed at the superior temporal gyrus and middle frontal gyrus. As for behavioral performance, no significant difference was observed between placebo and bupropion. CONCLUSIONS Our findings suggest that bupropion enhances the cerebral response to affective processing, especially positive emotional vocalizations, indicating a possible mechanism underlying the therapeutic effects for patients with depression.
Collapse
Affiliation(s)
- Tomoko Hama
- Department of Neuropsychiatry, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan; Faculty of Health Science Technology, Bunkyo Gakuin University, 2-4-1, Mukougaoka, Bunkyo-ku, Tokyo, 113-0023, Japan
| | - Michihiko Koeda
- Department of Neuropsychiatry, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan
| | - Yumiko Ikeda
- Department of Pharmacology, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan
| | - Amane Tateno
- Department of Neuropsychiatry, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan
| | - Tokuhiro Kawara
- Faculty of Health Science Technology, Bunkyo Gakuin University, 2-4-1, Mukougaoka, Bunkyo-ku, Tokyo, 113-0023, Japan
| | - Hidenori Suzuki
- Department of Pharmacology, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan
| | - Yoshiro Okubo
- Department of Neuropsychiatry, Graduate School of Medicine, Nippon Medical School, 1-1-5, Sendagi, Bunkyo-ku, Tokyo, 113-8603, Japan.
| |
Collapse
|
17
|
Shinkareva SV, Gao C, Wedell D. Audiovisual Representations of Valence: a Cross-study Perspective. ACTA ACUST UNITED AC 2020; 1:237-246. [DOI: 10.1007/s42761-020-00023-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Accepted: 10/22/2020] [Indexed: 01/25/2023]
|
18
|
Liang Y, Liu B. Cross-Subject Commonality of Emotion Representations in Dorsal Motion-Sensitive Areas. Front Neurosci 2020; 14:567797. [PMID: 33177977 PMCID: PMC7591793 DOI: 10.3389/fnins.2020.567797] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2020] [Accepted: 09/22/2020] [Indexed: 11/13/2022] Open
Abstract
Emotion perception is a crucial question in cognitive neuroscience and the underlying neural substrates have been the subject of intense study. One of our previous studies demonstrated that motion-sensitive areas are involved in the perception of facial expressions. However, it remains unclear whether emotions perceived from whole-person stimuli can be decoded from the motion-sensitive areas. In addition, if emotions are represented in the motion-sensitive areas, we may further ask whether the representations of emotions in the motion-sensitive areas can be shared across individual subjects. To address these questions, this study collected neural images while participants viewed emotions (joy, anger, and fear) from videos of whole-person expressions (contained both face and body parts) in a block-design functional magnetic resonance imaging (fMRI) experiment. Multivariate pattern analysis (MVPA) was conducted to explore the emotion decoding performance in individual-defined dorsal motion-sensitive regions of interest (ROIs). Results revealed that emotions could be successfully decoded from motion-sensitive ROIs with statistically significant classification accuracies for three emotions as well as positive versus negative emotions. Moreover, results from the cross-subject classification analysis showed that a person’s emotion representation could be robustly predicted by others’ emotion representations in motion-sensitive areas. Together, these results reveal that emotions are represented in dorsal motion-sensitive areas and that the representation of emotions is consistent across subjects. Our findings provide new evidence of the involvement of motion-sensitive areas in the emotion decoding, and further suggest that there exists a common emotion code in the motion-sensitive areas across individual subjects.
Collapse
Affiliation(s)
- Yin Liang
- Faculty of Information Technology, College of Computer Science and Technology, Beijing Artificial Intelligence Institute, Beijing University of Technology, Beijing, China
| | - Baolin Liu
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China
| |
Collapse
|
19
|
Kim MJ, Mattek AM, Shin J. Amygdalostriatal coupling underpins positive but not negative coloring of ambiguous affect. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2020; 20:949-960. [PMID: 32681315 PMCID: PMC7501244 DOI: 10.3758/s13415-020-00812-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Humans routinely integrate affective information from multiple sources. For example, we rarely interpret an emotional facial expression devoid of context. In this paper, we describe the neural correlates of an affective computation that involves integrating multiple sources, by leveraging the ambiguity and subtle feature-based valence signals found in surprised faces. Using functional magnetic resonance imaging, participants reported the valence of surprised faces modulated by positive or negative sentences. Amygdala activity corresponded to the valence value assigned to each contextually modulated face, with greater activity reflecting more negative ratings. Amygdala activity did not track the valence of the faces or sentences per se. Moreover, the amygdala was functionally coupled with the nucleus accumbens only during face trials preceded by positive contextual cues. These data suggest 1) valence-related amygdala activity reflects the integrated valence values rather than the valence values of each individual component, and 2) amygdalostriatal coupling underpins positive but not negative coloring of ambiguous affect.
Collapse
Affiliation(s)
- M Justin Kim
- Department of Psychology, Sungkyunkwan University, Seoul, South Korea.
| | - Alison M Mattek
- Department of Psychology, University of Oregon, Eugene, OR, USA
| | - Jin Shin
- Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO, USA
| |
Collapse
|
20
|
A coordinate-based meta-analysis of music-evoked emotions. Neuroimage 2020; 223:117350. [PMID: 32898679 DOI: 10.1016/j.neuroimage.2020.117350] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2020] [Revised: 08/04/2020] [Accepted: 09/01/2020] [Indexed: 12/19/2022] Open
Abstract
Since the publication of the first neuroscience study investigating emotion with music about two decades ago, the number of functional neuroimaging studies published on this topic has increased each year. This research interest is in part due to the ubiquity of music across cultures, and to music's power to evoke a diverse range of intensely felt emotions. To support a better understanding of the brain correlates of music-evoked emotions this article reports a coordinate-based meta-analysis of neuroimaging studies (n = 47 studies with n = 944 subjects). The studies employed a range of diverse experimental approaches (e.g., using music to evoke joy, sadness, fear, tension, frissons, surprise, unpleasantness, or feelings of beauty). The results of an activation likelihood estimation (ALE) indicate large clusters in a range of structures, including amygdala, anterior hippocampus, auditory cortex, and numerous structures of the reward network (ventral and dorsal striatum, anterior cingulate cortex, orbitofrontal cortex, secondary somatosensory cortex). The results underline the rewarding nature of music, the role of the auditory cortex as an emotional hub, and the role of the hippocampus in attachment-related emotions and social bonding.
Collapse
|
21
|
Reisch LM, Wegrzyn M, Woermann FG, Bien CG, Kissler J. Negative content enhances stimulus-specific cerebral activity during free viewing of pictures, faces, and words. Hum Brain Mapp 2020; 41:4332-4354. [PMID: 32633448 PMCID: PMC7502837 DOI: 10.1002/hbm.25128] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2020] [Revised: 06/16/2020] [Accepted: 06/24/2020] [Indexed: 01/25/2023] Open
Abstract
Negative visual stimuli have been found to elicit stronger brain activation than do neutral stimuli. Such emotion effects have been shown for pictures, faces, and words alike, but the literature suggests stimulus-specific differences regarding locus and lateralization of the activity. In the current functional magnetic resonance imaging study, we directly compared brain responses to passively viewed negative and neutral pictures of complex scenes, faces, and words (nouns) in 43 healthy participants (21 males) varying in age and demographic background. Both negative pictures and faces activated the extrastriate visual cortices of both hemispheres more strongly than neutral ones, but effects were larger and extended more dorsally for pictures, whereas negative faces additionally activated the superior temporal sulci. Negative words differentially activated typical higher-level language processing areas such as the left inferior frontal and angular gyrus. There were small emotion effects in the amygdala for faces and words, which were both lateralized to the left hemisphere. Although pictures elicited overall the strongest amygdala activity, amygdala response to negative pictures was not significantly stronger than to neutral ones. Across stimulus types, emotion effects converged in the left anterior insula. No gender effects were apparent, but age had a small, stimulus-specific impact on emotion processing. Our study specifies similarities and differences in effects of negative emotional content on the processing of different types of stimuli, indicating that brain response to negative stimuli is specifically enhanced in areas involved in processing of the respective stimulus type in general and converges across stimuli in the left anterior insula.
Collapse
Affiliation(s)
- Lea Marie Reisch
- Department of Psychology, University of Bielefeld, Bielefeld, Germany.,Epilepsy Centre Bethel, Krankenhaus Mara, Bielefeld, Germany
| | - Martin Wegrzyn
- Department of Psychology, University of Bielefeld, Bielefeld, Germany
| | | | | | - Johanna Kissler
- Department of Psychology, University of Bielefeld, Bielefeld, Germany
| |
Collapse
|
22
|
Gao C, Weber CE, Wedell DH, Shinkareva SV. An fMRI Study of Affective Congruence across Visual and Auditory Modalities. J Cogn Neurosci 2020; 32:1251-1262. [DOI: 10.1162/jocn_a_01553] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/14/2023]
Abstract
Abstract
Evaluating multisensory emotional content is a part of normal day-to-day interactions. We used fMRI to examine brain areas sensitive to congruence of audiovisual valence and their overlap with areas sensitive to valence. Twenty-one participants watched audiovisual clips with either congruent or incongruent valence across visual and auditory modalities. We showed that affective congruence versus incongruence across visual and auditory modalities is identifiable on a trial-by-trial basis across participants. Representations of affective congruence were widely distributed with some overlap with the areas sensitive to valence. Regions of overlap included bilateral superior temporal cortex and right pregenual anterior cingulate. The overlap between the regions identified here and in the emotion congruence literature lends support to the idea that valence may be a key determinant of affective congruence processing across a variety of discrete emotions.
Collapse
|
23
|
A study in affect: Predicting valence from fMRI data. Neuropsychologia 2020; 143:107473. [DOI: 10.1016/j.neuropsychologia.2020.107473] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2019] [Revised: 04/10/2020] [Accepted: 04/19/2020] [Indexed: 12/19/2022]
|
24
|
Sachs ME, Kaplan J, Habibi A. Echoing the emotions of others: empathy is related to how adults and children map emotion onto the body. Cogn Emot 2019; 33:1639-1654. [DOI: 10.1080/02699931.2019.1591938] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Affiliation(s)
- Matthew E. Sachs
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, USA
| | - Jonas Kaplan
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, USA
| | - Assal Habibi
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, USA
| |
Collapse
|
25
|
Gu J, Cao L, Liu B. Modality-general representations of valences perceived from visual and auditory modalities. Neuroimage 2019; 203:116199. [PMID: 31536804 DOI: 10.1016/j.neuroimage.2019.116199] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2018] [Revised: 08/31/2019] [Accepted: 09/14/2019] [Indexed: 01/29/2023] Open
Abstract
Valence is a dimension of emotion and can be either positive, negative, or neutral. Valences can be expressed through the visual and auditory modalities, and the valences of each modality can be conveyed by different types of stimuli (face, body, voice or music). This study focused on the modality-general representations of valences, that is, valence information can be shared across not only visual and auditory modalities but also different types of stimuli within each modality. Functional magnetic resonance imaging (fMRI) data were collected when subjects made affective judgment on silent videos (face and body) and audio clips (voice and music). The searchlight analysis helped to locate four areas that might be sensitive to the representations of modality-general valences, including the bilateral postcentral gyrus, left middle temporal gyrus (MTG) and right middle frontal gyrus (MFG). Further cross-modal classification based on multivoxel pattern analysis (MVPA) was performed as a validation analysis, which suggested that only the left postcentral gyrus could successfully distinguish three valences (positive versus negative and versus neutral: PvsNvs0) across different types of stimuli (face, body, voice or music), and the classification was also successful in left MTG across the stimuli types of face and body. The univariate analysis further found the valence-specific activation differences across stimulus types in MTG. Our study showed that the left postcentral gyrus was informative to valence representations, and extended the research about valence representation that the modality-general representation of valences across not only visual and auditory modalities but also different types of stimuli within each modality.
Collapse
Affiliation(s)
- Jin Gu
- College of Intelligence and Computing, Tianjin University, Tianjin, 300350, PR China
| | - Linjing Cao
- College of Intelligence and Computing, Tianjin University, Tianjin, 300350, PR China
| | - Baolin Liu
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, PR China.
| |
Collapse
|
26
|
Gao C, Baucom LB, Kim J, Wang J, Wedell DH, Shinkareva SV. Distinguishing abstract from concrete concepts in supramodal brain regions. Neuropsychologia 2019; 131:102-110. [PMID: 31175884 DOI: 10.1016/j.neuropsychologia.2019.05.032] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2018] [Revised: 01/18/2019] [Accepted: 05/31/2019] [Indexed: 11/24/2022]
Abstract
Concrete words have been shown to have a processing advantage over abstract words, yet theoretical accounts and neural correlates underlying the distinction between concrete and abstract concepts are still unresolved. In an fMRI study, participants performed a property verification task on abstract and concrete concepts. Property comparisons of concrete concepts were predominantly based on either visual or haptic features. Multivariate pattern analysis successfully distinguished between abstract and concrete stimulus comparisons at the whole brain level. Multivariate searchlight analyses showed that posterior and middle cingulate cortices contained information that distinguished abstract from concrete concepts regardless of feature dominance. These results support the view that supramodal convergence zones play an important role in representation of concrete and abstract concepts.
Collapse
Affiliation(s)
- Chuanji Gao
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA
| | - Laura B Baucom
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA
| | - Jongwan Kim
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA
| | - Jing Wang
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA
| | - Douglas H Wedell
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA
| | - Svetlana V Shinkareva
- Department of Psychology, Institute of Mind and Brain, University of South Carolina, Columbia, 29201, USA.
| |
Collapse
|
27
|
Pires FBC, Lacerda SS, Balardin JB, Portes B, Tobo PR, Barrichello CRC, Amaro E, Kozasa EH. Self-compassion is associated with less stress and depression and greater attention and brain response to affective stimuli in women managers. BMC WOMENS HEALTH 2018; 18:195. [PMID: 30482193 PMCID: PMC6258154 DOI: 10.1186/s12905-018-0685-y] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Accepted: 11/20/2018] [Indexed: 01/01/2023]
Abstract
BACKGROUND Women have been assuming more responsibilities and higher positions in major companies, which exposes them to high levels of stress. Higher perceived work stress is related to higher emotional reactivity. Difficulties with emotional regulation can lead to anxiety and mood disorders, which are more prevalent in women than men. Indeed, women leaders are more likely to experience emotional fatigue than men due to excessive empathy. Our aim was to evaluate the associations between self-compassion (SC) scores to depression symptoms, perceived stress and mindfulness, as well as with brain responses to high-arousal unpleasant and pleasant pictures from the International Affective Picture System (IAPS), as measured through functional Magnetic Resonance Imaging (fMRI) in women managers. METHODS Forty-six participants were selected for the study. All participants filled the Self Compassion Scale (SCS), Beck Depression Inventory (BDI), Mindful Awareness Attention Scale (MAAS) and the Perceived Stress Scale (PSS). After that they were scanned during an fMRI affective response paradigm. Correlation analysis were performed among these variables. RESULTS Our data suggest that women with higher SC scores respond to affective stimuli with higher activation of the precuneus (a brain region related to self-referential processing), lower levels of stress and depression and show greater attention in everyday activities. CONCLUSION SC may be an important characteristic for women leaders because of its association with higher sensitivity to emotional stimuli and mindfulness. These skills may allow them to be more aware of others while being less susceptible for stress and depression symptoms.
Collapse
Affiliation(s)
- Fernanda B C Pires
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil
| | - Shirley S Lacerda
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil.
| | - Joana B Balardin
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil
| | - Bruna Portes
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil
| | - Patrícia R Tobo
- Natura Cosméticos S.A, Rodovia Anhanguera km 30.5, Cajamar, SP, Brazil
| | | | - Edson Amaro
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil
| | - Elisa H Kozasa
- Hospital Israelita Albert Einstein, Av. Albert Einstein, 627/701, São Paulo, SP, Brazil
| |
Collapse
|
28
|
Gao C, Wedell DH, Green JJ, Jia X, Mao X, Guo C, Shinkareva SV. Temporal dynamics of audiovisual affective processing. Biol Psychol 2018; 139:59-72. [DOI: 10.1016/j.biopsycho.2018.10.001] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2018] [Revised: 08/28/2018] [Accepted: 10/01/2018] [Indexed: 11/16/2022]
|
29
|
Kim HC, Bandettini PA, Lee JH. Deep neural network predicts emotional responses of the human brain from functional magnetic resonance imaging. Neuroimage 2018; 186:607-627. [PMID: 30366076 DOI: 10.1016/j.neuroimage.2018.10.054] [Citation(s) in RCA: 20] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2017] [Revised: 08/15/2018] [Accepted: 10/21/2018] [Indexed: 10/28/2022] Open
Abstract
An artificial neural network with multiple hidden layers (known as a deep neural network, or DNN) was employed as a predictive model (DNNp) for the first time to predict emotional responses using whole-brain functional magnetic resonance imaging (fMRI) data from individual subjects. During fMRI data acquisition, 10 healthy participants listened to 80 International Affective Digital Sound stimuli and rated their own emotions generated by each sound stimulus in terms of the arousal, dominance, and valence dimensions. The whole-brain spatial patterns from a general linear model (i.e., beta-valued maps) for each sound stimulus and the emotional response ratings were used as the input and output for the DNNP, respectively. Based on a nested five-fold cross-validation scheme, the paired input and output data were divided into training (three-fold), validation (one-fold), and test (one-fold) data. The DNNP was trained and optimized using the training and validation data and was tested using the test data. The Pearson's correlation coefficients between the rated and predicted emotional responses from our DNNP model with weight sparsity optimization (mean ± standard error 0.52 ± 0.02 for arousal, 0.51 ± 0.03 for dominance, and 0.51 ± 0.03 for valence, with an input denoising level of 0.3 and a mini-batch size of 1) were significantly greater than those of DNN models with conventional regularization schemes including elastic net regularization (0.15 ± 0.05, 0.15 ± 0.06, and 0.21 ± 0.04 for arousal, dominance, and valence, respectively), those of shallow models including logistic regression (0.11 ± 0.04, 0.10 ± 0.05, and 0.17 ± 0.04 for arousal, dominance, and valence, respectively; average of logistic regression and sparse logistic regression), and those of support vector machine-based predictive models (SVMps; 0.12 ± 0.06, 0.06 ± 0.06, and 0.10 ± 0.06 for arousal, dominance, and valence, respectively; average of linear and non-linear SVMps). This difference was confirmed to be significant with a Bonferroni-corrected p-value of less than 0.001 from a one-way analysis of variance (ANOVA) and subsequent paired t-test. The weights of the trained DNNPs were interpreted and input patterns that maximized or minimized the output of the DNNPs (i.e., the emotional responses) were estimated. Based on a binary classification of each emotion category (e.g., high arousal vs. low arousal), the error rates for the DNNP (31.2% ± 1.3% for arousal, 29.0% ± 1.7% for dominance, and 28.6% ± 3.0% for valence) were significantly lower than those for the linear SVMP (44.7% ± 2.0%, 50.7% ± 1.7%, and 47.4% ± 1.9% for arousal, dominance, and valence, respectively) and the non-linear SVMP (48.8% ± 2.3%, 52.2% ± 1.9%, and 46.4% ± 1.3% for arousal, dominance, and valence, respectively), as confirmed by the Bonferroni-corrected p < 0.001 from the one-way ANOVA. Our study demonstrates that the DNNp model is able to reveal neuronal circuitry associated with human emotional processing - including structures in the limbic and paralimbic areas, which include the amygdala, prefrontal areas, anterior cingulate cortex, insula, and caudate. Our DNNp model was also able to use activation patterns in these structures to predict and classify emotional responses to stimuli.
Collapse
Affiliation(s)
- Hyun-Chul Kim
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul, 02841, Republic of Korea
| | - Peter A Bandettini
- Section on Functional Imaging Methods, Lab of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, 20892, USA
| | - Jong-Hwan Lee
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul, 02841, Republic of Korea.
| |
Collapse
|
30
|
Cao L, Xu J, Yang X, Li X, Liu B. Abstract Representations of Emotions Perceived From the Face, Body, and Whole-Person Expressions in the Left Postcentral Gyrus. Front Hum Neurosci 2018; 12:419. [PMID: 30405375 PMCID: PMC6200969 DOI: 10.3389/fnhum.2018.00419] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2018] [Accepted: 09/27/2018] [Indexed: 12/03/2022] Open
Abstract
Emotions can be perceived through the face, body, and whole-person, while previous studies on the abstract representations of emotions only focused on the emotions of the face and body. It remains unclear whether emotions can be represented at an abstract level regardless of all three sensory cues in specific brain regions. In this study, we used the representational similarity analysis (RSA) to explore the hypothesis that the emotion category is independent of all three stimulus types and can be decoded based on the activity patterns elicited by different emotions. Functional magnetic resonance imaging (fMRI) data were collected when participants classified emotions (angry, fearful, and happy) expressed by videos of faces, bodies, and whole-persons. An abstract emotion model was defined to estimate the neural representational structure in the whole-brain RSA, which assumed that the neural patterns were significantly correlated in within-emotion conditions ignoring the stimulus types but uncorrelated in between-emotion conditions. A neural representational dissimilarity matrix (RDM) for each voxel was then compared to the abstract emotion model to examine whether specific clusters could identify the abstract representation of emotions that generalized across stimulus types. The significantly positive correlations between neural RDMs and models suggested that the abstract representation of emotions could be successfully captured by the representational space of specific clusters. The whole-brain RSA revealed an emotion-specific but stimulus category-independent neural representation in the left postcentral gyrus, left inferior parietal lobe (IPL) and right superior temporal sulcus (STS). Further cluster-based MVPA revealed that only the left postcentral gyrus could successfully distinguish three types of emotions for the two stimulus type pairs (face-body and body-whole person) and happy versus angry/fearful, which could be considered as positive versus negative for three stimulus type pairs, when the cross-modal classification analysis was performed. Our study suggested that abstract representations of three emotions (angry, fearful, and happy) could extend from the face and body stimuli to whole-person stimuli and the findings of this study provide support for abstract representations of emotions in the left postcentral gyrus.
Collapse
Affiliation(s)
- Linjing Cao
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Junhai Xu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Xiaoli Yang
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China
| | - Xianglin Li
- Medical Imaging Research Institute, Binzhou Medical University, Yantai, China
| | - Baolin Liu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China.,State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Tsinghua University, Beijing, China
| |
Collapse
|
31
|
Eerola T, Vuoskoski JK, Peltola HR, Putkinen V, Schäfer K. An integrative review of the enjoyment of sadness associated with music. Phys Life Rev 2018; 25:100-121. [DOI: 10.1016/j.plrev.2017.11.016] [Citation(s) in RCA: 34] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2017] [Revised: 10/30/2017] [Accepted: 11/13/2017] [Indexed: 12/17/2022]
|
32
|
Sachs ME, Habibi A, Damasio A, Kaplan JT. Decoding the neural signatures of emotions expressed through sound. Neuroimage 2018; 174:1-10. [DOI: 10.1016/j.neuroimage.2018.02.058] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2017] [Revised: 02/23/2018] [Accepted: 02/27/2018] [Indexed: 12/15/2022] Open
|
33
|
Identification of task sets within and across stimulus modalities. Neuropsychologia 2018; 113:78-84. [DOI: 10.1016/j.neuropsychologia.2018.03.023] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2017] [Revised: 02/06/2018] [Accepted: 03/19/2018] [Indexed: 11/19/2022]
|
34
|
Sachs M, Habibi A, Damasio H. Reflections on music, affect, and sociality. PROGRESS IN BRAIN RESEARCH 2018; 237:153-172. [PMID: 29779733 DOI: 10.1016/bs.pbr.2018.03.009] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
Music is an important facet of and practice in human cultures, significantly related to its capacity to induce a range of intense and complex emotions. Studying the psychological and neurophysiological responses to music allows us to examine and uncover the neural mechanisms underlying the emotional impact of music. We provide an overview of different aspects of current research on how music listening produces emotions and the corresponding feelings, and consider the underlying neurophysiological mechanisms. We conclude with evidence to suggest that musical training may influence the ability to recognize the emotions of others.
Collapse
Affiliation(s)
- Matthew Sachs
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, United States
| | - Assal Habibi
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, United States
| | - Hanna Damasio
- Brain and Creativity Institute, University of Southern California, Los Angeles, CA, United States.
| |
Collapse
|
35
|
Miskovic V, Anderson AK. Modality general and modality specific coding of hedonic valence. Curr Opin Behav Sci 2018; 19:91-97. [PMID: 29967806 DOI: 10.1016/j.cobeha.2017.12.012] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Abstract
The pleasant or unpleasant qualities that attach to our perceptions help to determine whether we approach or avoid environmental stimuli, shaping their affordances. How do brains create this affective perceptual dimension? The traditional answer is that sensory areas serve only as conduits for external impressions that are then modulated by heteromodal limbic structures in subsequent phases. Here we raise the possibility that, in addition to these well established gain control effects, sensory systems might also have a more direct role in representing the pleasantness component of perception, as supported by several strands of recent brain imaging evidence. In conjunction with a shared valence code that is independent of its sensory origins, valence representations interleaved within sensory brain areas may support finer grained experiential distinctions between how things look, sound, feel, taste and smell good or bad to us, offering a higher dimensional space of evaluative discriminations.
Collapse
Affiliation(s)
- V Miskovic
- Department of Psychology, State University of New York at Binghamton, United States.,Center for Affective Science, State University of New York at Binghamton, United States
| | - A K Anderson
- Department of Human Development and Human Neuroscience Institute, Cornell University, United States
| |
Collapse
|
36
|
Gao C, Wedell DH, Kim J, Weber CE, Shinkareva SV. Modelling audiovisual integration of affect from videos and music. Cogn Emot 2017; 32:516-529. [DOI: 10.1080/02699931.2017.1320979] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Chuanji Gao
- Department of Psychology, University of South Carolina, Columbia, SC, USA
| | - Douglas H. Wedell
- Department of Psychology, University of South Carolina, Columbia, SC, USA
| | - Jongwan Kim
- Department of Psychology, University of South Carolina, Columbia, SC, USA
| | - Christine E. Weber
- Department of Psychology, University of South Carolina, Columbia, SC, USA
| | | |
Collapse
|