The effect of auditory valence on subsequent visual semantic processing

Emotion influences many cognitive processes and plays an important role in our daily life. Previous studies focused on the effects of arousal on subsequent cognitive processing, but the effect of valence on subsequent semantic processing is still not clear. The present study examined the effect of auditory valence on subsequent visual semantic processing when controlling for arousal. We used instrumental music clips varying in valence while matching in arousal to induce valence states and asked participants to make natural or man-made judgements on subsequent neutral objects. We found that positive and negative valences similarly impaired subsequent semantic processing compared with neutral valence. The linear ballistic accumulator model analyses showed that the valence effects can be attributed to drift rate differences, suggesting that the effects are likely related to attentional selection. Our findings are consistent with a motivated attention model, indicating comparable attentional capture by both positive and negative valences in modulating subsequent cognitive processes.

How do valence and meaning interact? The contribution of semantic control

The hub-and-spoke model of semantic cognition proposes that conceptual representations in a heteromodal 'hub' interact with and emerge from modality-specific features or 'spokes', including valence (whether a concept is positive or negative), along with visual and auditory features. As a result, valence congruency might facilitate our ability to link words conceptually. Semantic relatedness may similarly affect explicit judgements about valence. Moreover, conflict between meaning and valence may recruit semantic control processes. Here we tested these predictions using two-alternative forced-choice tasks, in which participants matched a probe word to one of two possible target words, based on either global meaning or valence. Experiment 1 examined timed responses in healthy young adults, while Experiment 2 examined decision accuracy in semantic aphasia patients with impaired controlled semantic retrieval following left hemisphere stroke. Across both experiments, semantically related targets facilitated valence matching, while related distractors impaired performance. Valence congruency was also found to facilitate semantic decision-making. People with semantic aphasia showed impaired valence matching and had particular difficulty when semantically related distractors were presented, suggesting that the selective retrieval of valence information relies on semantic control processes. Taken together, the results are consistent with the hypothesis that automatic access to the global meaning of written words affects the processing of valence, and that the valence of words is also retrieved even when this feature is task-irrelevant, affecting the efficiency of global semantic judgements.

Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience

Determining and decoding emotional brain processes under ecologically valid conditions remains a key challenge in affective neuroscience. The current functional Magnetic Resonance Imaging (fMRI) based emotion decoding studies are mainly based on brief and isolated episodes of emotion induction, while sustained emotional experience in naturalistic environments that mirror daily life experiences are scarce. Here we used 12 different 10-minute movie clips as ecologically valid emotion-evoking procedures in n = 52 individuals to explore emotion-specific fMRI functional connectivity (FC) profiles on the whole-brain level at high spatial resolution (432 parcellations including cortical and subcortical structures). Employing machine-learning based decoding and cross validation procedures allowed to investigate FC profiles contributing to classification that can accurately distinguish sustained happiness and sadness and that generalize across subjects, movie clips, and parcellations. Both functional brain network-based and subnetwork-based emotion classification results suggested that emotion manifests as distributed representation of multiple networks, rather than a single functional network or subnetwork. Further, the results showed that the Visual Network (VN) and Default Mode Network (DMN) associated functional networks, especially VN-DMN, exhibited a strong contribution to emotion classification. To further estimate the temporal accumulative effect of naturalistic long-term movie-based video-evoking emotions, we divided the 10-min episode into three stages: early stimulation (1∼200 s), middle stimulation (201∼400 s), and late stimulation (401∼600 s) and examined the emotion classification performance at different stimulation stages. We found that the late stimulation contributes most to the classification (accuracy=85.32%, F1-score=85.62%) compared to early and middle stimulation stages, implying that continuous exposure to emotional stimulation can lead to more intense emotions and further enhance emotion-specific distinguishable representations. The present work demonstrated that sustained happiness and sadness under naturalistic conditions are presented in emotion-specific network profiles and these expressions may play different roles in the generation and modulation of emotions. These findings elucidated the importance of network level adaptations for sustained emotional experiences during naturalistic contexts and open new venues for imaging network level contributions under naturalistic conditions.

Effects of emotional congruency and task complexity on decision-making

The heuristic approach to decision-making holds that the selection process becomes more efficient when part of the information available is ignored. One element involved in selecting information is emotional valence. If emotional congruency is related to simplified decision-making strategies, then the interaction of this factor with task complexity should exist. The present study explored how factors of this nature influence decision-making efficiency. We hypothesized that emotional congruency would have a positive effect on task execution and that the magnitude of that effect would increase with greater task complexity because in that condition the amount of information to be processed is greater, meaning that a heuristic approach to the problem would be more efficient. We design a decision in browser decision-making task in which participants had to select emotional images to gain points. Depending on the correlation between emotional valence and in-task image value, we defined three emotional congruency conditions: direct, null, and inverse. Our results show that distinct types of emotional congruency have differential effects on behavior. While direct congruency-enhanced overall decision-making performance, inverse congruency interacted with task complexity to modify the pace at which task feedback affected behavior.

Sound-modulations of visual motion perception implicate the cortico-vestibular brain

A widely used example of the intricate (yet poorly understood) intertwining of multisensory signals in the brain is the audiovisual bounce inducing effect (ABE). This effect presents two identical objects moving along the azimuth with uniform motion and towards opposite directions. The perceptual interpretation of the motion is ambiguous and is modulated if a transient (sound) is presented in coincidence with the point of overlap of the two objects' motion trajectories. This phenomenon has long been written-off to simple attentional or decision-making mechanisms, although the neurological underpinnings for the effect are not well understood. Using behavioural metrics concurrently with event-related fMRI, we show that sound-induced modulations of motion perception can be further modulated by changing motion dynamics of the visual targets. The phenomenon engages the posterior parietal cortex and the parieto-insular-vestibular cortical complex, with a close correspondence of activity in these regions with behaviour. These findings suggest that the insular cortex is engaged in deriving a probabilistic perceptual solution through the integration of multisensory data.

Individualized video recommendation modulates functional connectivity between large scale networks

With the emergence of AI‐powered recommender systems and their extensive use in the video streaming service, questions and concerns also arise. Why can recommended video content continuously capture users' attention? What is the impact of long‐term exposure to personalized video content on one's behaviors and brain functions? To address these questions, we designed an fMRI experiment presenting participants with personally recommended videos and generally recommended ones. To examine how large‐scale networks were modulated by personalized video content, graph theory analysis was applied to investigate the interaction between seven networks, including the ventral and dorsal attention networks (VAN, DAN), frontal–parietal network (FPN), salience network (SN), and three subnetworks of default mode network (dorsal medial prefrontal (dMPFC), Core, and medial temporal lobe (MTL)). Our results showed that viewing nonpersonalized video content mainly enhanced the connectivity in the DAN‐FPN‐Core pathway, whereas viewing personalized ones increased not only the connectivity in this pathway but also the DAN‐VAN‐dMPFC pathway. In addition, both personalized and nonpersonalized short videos decreased the couplings between SN and VAN as well as between two DMN subsystems, Core and MTL. Collectively, these findings uncovered distinct patterns of network interactions in response to short videos and provided insights into potential neural mechanisms by which human behaviors are biased by personally recommended content.

Modality-general and modality-specific audiovisual valence processing

A fundamental question in affective neuroscience is whether there is a common hedonic system for valence processing independent of modality, or there are distinct neural systems for different modalities. To address this question, we used both region of interest and whole-brain representational similarity analyses on functional magnetic resonance imaging data to identify modality-general and modality-specific brain areas involved in valence processing across visual and auditory modalities. First, region of interest analyses showed that the superior temporal cortex was associated with both modality-general and auditory-specific models, while the primary visual cortex was associated with the visual-specific model. Second, the whole-brain searchlight analyses also identified both modality-general and modality-specific representations. The modality-general regions included the superior temporal, medial superior frontal, inferior frontal, precuneus, precentral, postcentral, supramarginal, paracentral lobule and middle cingulate cortices. The modality-specific regions included both perceptual cortices and higher-order brain areas. The valence representations derived from individualized behavioral valence ratings were consistent with these results. Together, these findings suggest both modality-general and modality-specific representations of valence.

Supracategorical fear information revealed by aversively conditioning multiple categories

Fear-generalization is a critical function for survival, in which an organism extracts information from a specific instantiation of a threat (e.g., the western diamondback rattlesnake in my front yard on Sunday) and learns to fear - and accordingly respond to - pertinent higher-order information (e.g., snakes live in my yard). Previous work investigating fear-conditioning in humans has used functional magnetic resonance imaging (fMRI) to demonstrate that activity patterns representing stimuli from an aversively-conditioned category (CS+) are more similar to each other than those of a neutral category (CS-). Here we used fMRI and multiple aversively-conditioned categories to ask whether we would find only similarity increases within the CS+ categories or also similarity increases between the CS+ categories. Using representational similarity analysis, we correlated several models to activity patterns underlying different brain regions and found that, following fear-conditioning, between-category and within-category similarity increased for the CS+ categories in the insula, superior frontal gyrus (SFG), and the right temporal pole. When specifically investigating fear-generalization, these between- and within-category effects were detected in the SFG. These results advance prior pattern-based neuroimaging work by exploring the effect of aversively-conditioning multiple categories and indicate an extended role for such regions in potentially representing supracategorical information during fear-learning.