Decoding Neural Representations of Affective Scenes in Retinotopic Visual Cortex

Moving toward reality: Electrocortical reactivity to naturalistic multimodal emotional videos

While previous research has investigated the effects of emotional videos on peripheral physiological measures and conscious experience, this study extends the research to include electrocortical measures, specifically the steady-state visual-evoked potential (ssVEP). A carefully curated set of 45 videos, designed to represent a wide range of emotional and neutral content, were presented with a flickering border. The videos featured a continuous single-shot perspective, natural soundtrack, and excluded elements associated with professional films, to enhance realism. The results demonstrate a consistent reduction in ssVEP amplitude during emotional videos which strongly correlates with the rated emotional intensity of the clips. This suggests that narrative audiovisual stimuli have the potential to track dynamic emotional processing in the cortex, providing new avenues for research in affective neuroscience. The findings highlight the potential of using realistic video stimuli to investigate how the human brain processes emotional events in a paradigm that increases ecological validity. Future studies can further develop this paradigm by expanding the video set, targeting specific cortical networks, and manipulating narrative predictability. Overall, this study establishes a foundation for investigating emotional perception using realistic video stimuli and has the potential to expand our understanding of real-world emotional processing in the human brain.

A systems identification approach using Bayes factors to deconstruct the brain bases of emotion regulation

Cognitive reappraisal is fundamental to cognitive therapies and everyday emotion regulation. Analyses using Bayes factors and an axiomatic systems identification approach identified four reappraisal-related components encompassing distributed neural activity patterns across two independent functional magnetic resonance imaging (fMRI) studies (n = 182 and n = 176): (1) an anterior prefrontal system selectively involved in cognitive reappraisal; (2) a fronto-parietal-insular system engaged by both reappraisal and emotion generation, demonstrating a general role in appraisal; (3) a largely subcortical system activated during negative emotion generation but unaffected by reappraisal, including amygdala, hypothalamus and periaqueductal gray; and (4) a posterior cortical system of negative emotion-related regions downregulated by reappraisal. These systems covaried with individual differences in reappraisal success and were differentially related to neurotransmitter binding maps, implicating cannabinoid and serotonin systems in reappraisal. These findings challenge 'limbic'-centric models of reappraisal and provide new systems-level targets for assessing and enhancing emotion regulation.

Resting heart rate variability is associated with neural adaptation when repeatedly exposed to emotional stimuli

Higher heart rate variability (HRV) at rest is associated with better emotion regulation ability. While the neurovisceral integration model explains this by postulating that HRV can index how the brain adaptively modulates responses to emotional stimuli, neuroimaging studies directly supporting this idea are scarce. We examined the neural correlates of regulating negative and positive emotion in relation to resting HRV based on the neuroimaging and heart rate data of one hundred young adults. The results showed that those with higher HRV better recruit the medial prefrontal cortex while intensifying positive compared to negative emotion. We also examined how individual differences in resting HRV are associated with adjusting brain activity to repeated emotional stimuli. During repeated viewing of emotional images, subjects with higher resting HRV better reduced activity in the medial prefrontal cortex, posterior cingulate gyrus, and angular gyrus, most of which overlapped with the default mode network. This HRV-DMN association was observed during passively viewing emotional images rather than during actively regulating emotion. While the regulating trials can better detect task-induced changes, the viewing trials might approximate resting state, better revealing individual differences. These findings suggest two possibilities: people with higher resting HRV might have a tendency to spontaneously engage with emotion regulation or possess a trait helping emotional arousal fade away.

Additive Effects of Monetary Loss and Positive Emotion in the Human Brain

In many real-life scenarios, our decisions could lead to multiple outcomes that conflict with value. Hence, an appropriate neural representation of the net experienced value of conflicting outcomes, which play a crucial role in guiding future decisions, is critical for adaptive behavior. As some recent functional neuroimaging work has primarily focused on the concurrent processing of monetary gains and aversive information, very little is known regarding the integration of conflicting value signals involving monetary losses and appetitive information in the human brain. To address this critical gap, we conducted a functional MRI study involving healthy human male participants to examine the nature of integrating positive emotion and monetary losses. We employed a novel experimental design where the valence (positive or neutral) of an emotional stimulus indicated the type of outcome (loss or no loss) in a choice task. Specifically, we probed two plausible integration patterns while processing conflicting value signals involving positive emotion and monetary losses: interactive versus additive. We found overlapping main effects of positive (vs neutral) emotion and loss (vs no loss) in multiple brain regions, including the ventromedial prefrontal cortex, striatum, and amygdala, notably with a lack of evidence for interaction. Thus, our findings revealed the additive integration pattern of monetary loss and positive emotion outcomes, suggesting that the experienced value of the monetary loss was not modulated by the valence of the image signaling those outcomes. These findings contribute to our limited understanding of the nature of integrating conflicting outcomes in the healthy human brain with potential clinical relevance.

Neural patterns associated with mixed valence feelings differ in consistency and predictability throughout the brain

Mixed feelings, the simultaneous presence of feelings with positive and negative valence, remain an understudied topic. They pose a specific set of challenges due to individual variation, and their investigation requires analtyic approaches focusing on individually self-reported states. We used functional magnetic resonance imaging (fMRI) to scan 27 subjects watching an animated short film chosen to induce bittersweet mixed feelings. The same subjects labeled when they had experienced positive, negative, and mixed feelings. Using hidden-Markov models, we found that various brain regions could predict the onsets of new feeling states as determined by self-report. The ability of the models to identify these transitions suggests that these states may exhibit unique and consistent neural signatures. We next used the subjects' self-reports to evaluate the spatiotemporal consistency of neural patterns for positive, negative, and mixed states. The insula had unique and consistent neural signatures for univalent states, but not for mixed valence states. The anterior cingulate and ventral medial prefrontal cortex had consistent neural signatures for both univalent and mixed states. This study is the first to demonstrate that subjectively reported changes in feelings induced by naturalistic stimuli can be predicted from fMRI and the first to show direct evidence for a neurally consistent representation of mixed feelings.

Adaptive-to-maladaptive gradient of emotion regulation tendencies are embedded in the functional-structural hybrid connectome

BACKGROUND

Emotion regulation tendencies are well-known transdiagnostic markers of psychopathology, but their neurobiological foundations have mostly been examined within the theoretical framework of cortical-subcortical interactions.

METHODS

We explored the connectome-wide neural correlates of emotion regulation tendencies using functional and diffusion magnetic resonance images of healthy young adults (N = 99; age 20-30; 28 females). We first tested the importance of considering both the functional and structural connectome through intersubject representational similarity analyses. Then, we employed a canonical correlation analysis between the functional-structural hybrid connectome and 23 emotion regulation strategies. Lastly, we sought to externally validate the results on a transdiagnostic adolescent sample (N = 93; age 11-19; 34 females).

RESULTS

First, interindividual similarity of emotion regulation profiles was significantly correlated with interindividual similarity of the functional-structural hybrid connectome, more so than either the functional or structural connectome. Canonical correlation analysis revealed that an adaptive-to-maladaptive gradient of emotion regulation tendencies mapped onto a specific configuration of covariance within the functional-structural hybrid connectome, which primarily involved functional connections in the motor network and the visual networks as well as structural connections in the default mode network and the subcortical-cerebellar network. In the transdiagnostic adolescent dataset, stronger functional signatures of the found network were associated with higher general positive affect through more frequent use of adaptive coping strategies.

CONCLUSIONS

Taken together, our study illustrates a gradient of emotion regulation tendencies that is best captured when simultaneously considering the functional and structural connections across the whole brain.

Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that (1) in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and (2) lesioning these neurons by setting their output to 0 or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.

Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and lesioning these neurons by setting their output to zero or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.

In the face of ambiguity: intrinsic brain organization in development predicts one's bias toward positivity or negativity

Exacerbated negativity bias, including in responses to ambiguity, represents a common phenotype of internalizing disorders. Individuals differ in their propensity toward positive or negative appraisals of ambiguity. This variability constitutes one's valence bias, a stable construct linked to mental health. Evidence suggests an initial negativity in response to ambiguity that updates via regulatory processes to support a more positive bias. Previous work implicates the amygdala and prefrontal cortex, and regions of the cingulo-opercular system, in this regulatory process. Nonetheless, the neurodevelopmental origins of valence bias remain unclear. The current study tests whether intrinsic brain organization predicts valence bias among 119 children and adolescents (6 to 17 years). Using whole-brain resting-state functional connectivity, a machine-learning model predicted valence bias (r = 0.20, P = 0.03), as did a model restricted to amygdala and cingulo-opercular system features (r = 0.19, P = 0.04). Disrupting connectivity revealed additional intra-system (e.g. fronto-parietal) and inter-system (e.g. amygdala to cingulo-opercular) connectivity important for prediction. The results highlight top-down control systems and bottom-up perceptual processes that influence valence bias in development. Thus, intrinsic brain organization informs the neurodevelopmental origins of valence bias, and directs future work aimed at explicating related internalizing symptomology.

Decoding fMRI data with support vector machines and deep neural networks

BACKGROUND

Multivoxel pattern analysis (MVPA) examines fMRI activation patterns associated with different cognitive conditions. Support vector machines (SVMs) are the predominant method in MVPA. While SVM is intuitive and easy to apply, it is mainly suitable for analyzing data that are linearly separable. Convolutional neural networks (CNNs) are known to have the ability to approximate nonlinear relationships. Applications of CNN to fMRI data are beginning to appear with increasing frequency, but our understanding of the similarities and differences between CNN models and SVM models is limited.

NEW METHOD

We compared the two methods when they are applied to the same datasets. Two datasets were considered: (1) fMRI data collected from participants during a cued visual spatial attention task and (2) fMRI data collected from participants viewing natural images containing varying degrees of affective content.

RESULTS

We found that (1) both SVM and CNN are able to achieve above-chance decoding accuracies for attention control and emotion processing in both the primary visual cortex and the whole brain, (2) the CNN decoding accuracies are consistently higher than that of the SVM, (3) the SVM and CNN decoding accuracies are generally not correlated, and (4) the heatmaps derived from SVM and CNN are not significantly overlapping.

COMPARISON WITH EXISTING METHODS

By comparing SVM and CNN we pointed out the similarities and differences between the two methods.

CONCLUSIONS

SVM and CNN rely on different neural features for classification. Applying both to the same data may yield a more comprehensive understanding of neuroimaging data.

Decoding fMRI Data: A Comparison Between Support Vector Machines and Deep Neural Networks

Multivoxel pattern analysis (MVPA) examines the differences in fMRI activation patterns associated with different cognitive conditions and provides information not possible with the conventional univariate analysis. Support vector machines (SVMs) are the predominant machine learning method in MVPA. SVMs are intuitive and easy to apply. The limitation is that it is a linear method and mainly suitable for analyzing data that are linearly separable. Convolutional neural networks (CNNs), a class of AI models originally developed for object recognition, are known to have the ability to approximate nonlinear relationships. CNNs are rapidly becoming an alternative to SVMs. The purpose of this study is to compare the two methods when they are applied to the same datasets. Two datasets were considered: (1) fMRI data collected from participants during a cued visual spatial attention task (the attention dataset) and (2) fMRI data collected from participants viewing natural images containing varying degrees of affective content (the emotion dataset). We found that (1) both SVM and CNN are able to achieve above chance level decoding accuracies for attention control and emotion processing in both the primary visual cortex and the whole brain with, (2) the CNN decoding accuracies are consistently higher than that of the SVM, (3) the SVM and CNN decoding accuracies are generally not correlated with each other, and (4) the heatmaps derived from SVM and CNN are not significantly overlapping. These results suggest that (1) there are both linearly separable features and nonlinearly separable features in fMRI data that distinguish cognitive conditions and (2) applying both SVM and CNN to the same data may yield a more comprehensive understanding of neuroimaging data.

Key points

We compared the performance and characteristics of SVM and CNN, two major methods in MVPA analysis of neuroimaging data, by applying them to the same two fMRI datasets.Both SVM and CNN achieved decoding accuracies above chance level for both datasets in the chosen ROIs and the CNN decoding accuracies were consistently higher than those of SVM.The heatmaps derived from SVM and CNN, which assess the contribution of voxels or brain regions to MVPA decoding performance, showed no significant overlap, providing evidence that the two methods depend on distinct brain activity patterns for decoding cognitive conditions.

Altered brain function in patients with acrophobia: A voxel-wise degree centrality analysis

AIM

To explore the local spontaneous neural activity and whole-brain functional connectivity patterns in the resting brain of acrophobia patients.

METHODS

50 patients with acrophobia and 47 healthy controls were selected for this study. All participants underwent resting-state MRI scans after enrollment. The imaging data were then analyzed using a voxel-based degree centrality (DC) method, and seed-based functional connectivity (FC) correlation analysis was used to explore the correlation between abnormal functional connectivity and clinical symptom scales in acrophobia. The severity of symptoms was evaluated using self-report and behavioral measures.

RESULTS

Compared to controls, acrophobia patients showed higher DC in the right cuneus and left middle occipital gyrus and significantly lower DC in the right cerebellum and left orbitofrontal cortex (p < 0.01, GRF corrected). Additionally, there were negative correlations between the acrophobia questionnaire avoidance (AQ- Avoidance) scores and right cerebellum-left perirhinal cortex FC (r = -0.317, p = 0.025) and between scores of the 7-item generalized anxiety disorder scale and left middle occipital gyrus-right cuneus FC (r = -0.379, p = 0.007). In the acrophobia group, there was a positive correlation between behavioral avoidance scale and right cerebellum-right cuneus FC (r = 0.377, p = 0.007).

CONCLUSIONS

The findings indicated that there are local abnormalities in spontaneous neural activity and functional connectivity in the visual cortex, cerebellum, and orbitofrontal cortex in patients with acrophobia.

Recognition of Emotional and Neutral Visual Scenes in Carriers of the MAOA, COMT, DRD4, and 5HT2A Gene Polymorphisms

Background

It is known that some genes regulate neurochemical metabolism, and their polymorphisms affect cognitive performance, including the ability to categorize emotionally significant information.

Objective

The aim of our study was to analyze the recognition of emotional and neutral visual scenes in carriers of different polymorphic variants of the MAOA, COMT, DRD4, and 5HT2A genes.

Design

The study sample consisted of 87 university students (Caucasians, women 63%, average age 20.4±2.6 years). The genotypes of the COMT, 5HT2A, and DRD4 genes were determined by polymerase chain reaction. Agarose gel electrophoresis was used to determine the number of tandem repeats of the MAOA gene. Three hundred sixty (360) photographic images of scenes of different emotional valence (positive, negative, and neutral - 120 images for each category) were used as stimuli. These images were classified by expert assessments. The images were presented in a random sequence. The exposure time was 700 ms. The research participants were asked to determine the emotional valence of each scene.

Results

We found that only the COMT gene genotype affected the recognition of emotional and neutral visual scenes. Carriers of the COMT Val/Val genotype, which causes dopamine to stay in the synaptic space for a shorter time, are better in recognizing and demonstrate higher sensitivity to the emotional content of scenes. Carriers of the Val/Met genotype demonstrated the worst ability to differentiate the emotional valence of visual scenes.

Conclusion

This study has shown that the length of stay of monoamines in the synaptic space regulated by the COMT gene affects the recognition of emotional visual information.

Decoding the temporal dynamics of affective scene processing

Natural images containing affective scenes are used extensively to investigate the neural mechanisms of visual emotion processing. Functional fMRI studies have shown that these images activate a large-scale distributed brain network that encompasses areas in visual, temporal, and frontal cortices. The underlying spatial and temporal dynamics, however, remain to be better characterized. We recorded simultaneous EEG-fMRI data while participants passively viewed affective images from the International Affective Picture System (IAPS). Applying multivariate pattern analysis to decode EEG data, and representational similarity analysis to fuse EEG data with simultaneously recorded fMRI data, we found that: (1) ∼80 ms after picture onset, perceptual processing of complex visual scenes began in early visual cortex, proceeding to ventral visual cortex at ∼100 ms, (2) between ∼200 and ∼300 ms (pleasant pictures: ∼200 ms; unpleasant pictures: ∼260 ms), affect-specific neural representations began to form, supported mainly by areas in occipital and temporal cortices, and (3) affect-specific neural representations were stable, lasting up to ∼2 s, and exhibited temporally generalizable activity patterns. These results suggest that affective scene representations in the brain are formed temporally in a valence-dependent manner and may be sustained by recurrent neural interactions among distributed brain areas.

Neurostructural correlates of dispositional self-compassion

Self-compassion is an important emotion regulation strategy predicting positive psychological health and fewer psychopathological problems, but little is known about its structural neural basis. In the current study, we investigated the neurostructural correlates of dispositional self-compassion and its components using voxel-based morphometry (VBM). We found that self-compassion was inversely correlated with gray matter volume (GMV) in the left dorsolateral prefrontal cortex (DLPFC), which was primarily driven by the reduced self-judgment component. We also found that the mindfulness component was associated with greater GMV in the dorsomedial prefrontal cortex/anterior cingulate cortex and the left supplementary motor area, while the isolation and the over-identification components were both correlated with greater GMV in the right inferior temporal gyrus, and over-identification additionally related to less GMV in visual areas. Our findings suggest that dispositional self-compassion and its components are associated with brain structure in regions involved in emotion regulation, self-referential and emotion processing, with implications for the cognitive and neural mechanisms of self-compassion as well as those underlying the effects of self-compassion on its health outcomes.