The brain basis of audiovisual affective processing: Evidence from a coordinate-based activation likelihood estimation meta-analysis

The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS): A database for emotional auditory-only, visual-only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities

We describe JAVMEPS, an audiovisual (AV) database for emotional voice and dynamic face stimuli, with voices varying in emotional intensity. JAVMEPS includes 2256 stimulus files comprising (A) recordings of 12 speakers, speaking four bisyllabic pseudowords with six naturalistic induced basic emotions plus neutral, in auditory-only, visual-only, and congruent AV conditions. It furthermore comprises (B) caricatures (140%), original voices (100%), and anti-caricatures (60%) for happy, fearful, angry, sad, disgusted, and surprised voices for eight speakers and two pseudowords. Crucially, JAVMEPS contains (C) precisely time-synchronized congruent and incongruent AV (and corresponding auditory-only) stimuli with two emotions (anger, surprise), (C1) with original intensity (ten speakers, four pseudowords), (C2) and with graded AV congruence (implemented via five voice morph levels, from caricatures to anti-caricatures; eight speakers, two pseudowords). We collected classification data for Stimulus Set A from 22 normal-hearing listeners and four cochlear implant users, for two pseudowords, in auditory-only, visual-only, and AV conditions. Normal-hearing individuals showed good classification performance (McorrAV = .59 to .92), with classification rates in the auditory-only condition ≥ .38 correct (surprise: .67, anger: .51). Despite compromised vocal emotion perception, CI users performed above chance levels of .14 for auditory-only stimuli, with best rates for surprise (.31) and anger (.30). We anticipate JAVMEPS to become a useful open resource for researchers into auditory emotion perception, especially when adaptive testing or calibration of task difficulty is desirable. With its time-synchronized congruent and incongruent stimuli, JAVMEPS can also contribute to filling a gap in research regarding dynamic audiovisual integration of emotion perception via behavioral or neurophysiological recordings.

Association between gaming disorder and regional homogeneity in highly involved male adult gamers: A pilot resting-state fMRI study

BACKGROUND

Gaming behavior can induce cerebral changes that may be related to the neurobiological features of gaming disorder (GD). Additionally, individuals with higher levels of depression or impulsivity are more likely to experience GD. Therefore, the present pilot study explored potential neurobiological correlates of GD in the context of depression and impulsivity, after accounting for video gaming behavior.

METHODS

Using resting-state functional magnetic resonance imaging (fMRI), a cross-sectional study was conducted with 35 highly involved male adult gamers to examine potential associations between GD severity and regional homogeneity (ReHo) in the entire brain. A mediation model was used to test the role of ReHo in the possible links between depression/impulsivity and GD severity.

RESULTS

Individuals with greater GD severity showed increased ReHo in the right Heschl's gyrus and decreased ReHo in the right hippocampus (rHip). Furthermore, depression and impulsivity were negatively correlated with ReHo in the rHip, respectively. More importantly, ReHo in the rHip was found to mediate the associations between depression/impulsivity and GD.

CONCLUSIONS

These preliminary findings suggest that GD severity is related to ReHo in brain regions associated with learning/memory/mood and auditory function. Higher levels of depression or impulsivity may potentiate GD through the functional activity of the hippocampus. Our findings advance our understanding of the neurobiological differences behind GD symptoms in highly involved gamers.

Anterior cingulate glutamate levels associate with functional activation and connectivity during sensory integration in schizophrenia: a multimodal 1H-MRS and fMRI study

BACKGROUND

Glutamatergic dysfunction has been implicated in sensory integration deficits in schizophrenia, yet how glutamatergic function contributes to behavioural impairments and neural activities of sensory integration remains unknown.

METHODS

Fifty schizophrenia patients and 43 healthy controls completed behavioural assessments for sensory integration and underwent magnetic resonance spectroscopy (MRS) for measuring the anterior cingulate cortex (ACC) glutamate levels. The correlation between glutamate levels and behavioural sensory integration deficits was examined in each group. A subsample of 20 pairs of patients and controls further completed an audiovisual sensory integration functional magnetic resonance imaging (fMRI) task. Blood Oxygenation Level Dependent (BOLD) activation and task-dependent functional connectivity (FC) were assessed based on fMRI data. Full factorial analyses were performed to examine the Group-by-Glutamate Level interaction effects on fMRI measurements (group differences in correlation between glutamate levels and fMRI measurements) and the correlation between glutamate levels and fMRI measurements within each group.

RESULTS

We found that schizophrenia patients exhibited impaired sensory integration which was positively correlated with ACC glutamate levels. Multimodal analyses showed significantly Group-by-Glutamate Level interaction effects on BOLD activation as well as task-dependent FC in a 'cortico-subcortical-cortical' network (including medial frontal gyrus, precuneus, ACC, middle cingulate gyrus, thalamus and caudate) with positive correlations in patients and negative in controls.

CONCLUSIONS

Our findings indicate that ACC glutamate influences neural activities in a large-scale network during sensory integration, but the effects have opposite directionality between schizophrenia patients and healthy people. This implicates the crucial role of glutamatergic system in sensory integration processing in schizophrenia.

Pupil dilation reflects the dynamic integration of audiovisual emotional speech

Emotional speech perception is a multisensory process. When speaking with an individual we concurrently integrate the information from their voice and face to decode e.g., their feelings, moods, and emotions. However, the physiological reactions-such as the reflexive dilation of the pupil-associated to these processes remain mostly unknown. That is the aim of the current article, to investigate whether pupillary reactions can index the processes underlying the audiovisual integration of emotional signals. To investigate this question, we used an algorithm able to increase or decrease the smiles seen in a person's face or heard in their voice, while preserving the temporal synchrony between visual and auditory channels. Using this algorithm, we created congruent and incongruent audiovisual smiles, and investigated participants' gaze and pupillary reactions to manipulated stimuli. We found that pupil reactions can reflect emotional information mismatch in audiovisual speech. In our data, when participants were explicitly asked to extract emotional information from stimuli, the first fixation within emotionally mismatching areas (i.e., the mouth) triggered pupil dilation. These results reveal that pupil dilation can reflect the dynamic integration of audiovisual emotional speech and provide insights on how these reactions are triggered during stimulus perception.

Your smile won't affect me: Association between childhood maternal antipathy and adult neural reward function in a transdiagnostic sample

Aberrant activation in the ventral striatum (VS) during reward anticipation may be a key mechanism linking adverse childhood experiences (ACE) to transdiagnostic psychopathology. This study aimed to elucidate whether retrospectively reported ACE, specifically maternal antipathy, relate to monetary and social reward anticipation in a transdiagnostic adult sample. A cross-sectional neuroimaging study was conducted in 118 participants with varying levels of ACE, including 25 participants with posttraumatic stress disorder (PTSD), 32 with major depressive disorder (MDD), 29 with somatic symptom disorder (SSD), and 32 healthy volunteers (HVs). Participants underwent functional magnetic resonance imaging during a monetary and social incentive delay task, and completed a self-report measure of ACE, including maternal antipathy. Neural correlates of monetary and social reward anticipation and their association with ACE, particularly maternal antipathy, were analyzed. Participants showed elevated activation in brain regions underlying reward processing, including the VS, only while anticipating social, but not monetary rewards. Participants reporting higher levels of maternal antipathy exhibited reduced activation in the brain reward network, including the VS, only during social, but not monetary reward anticipation. Group affiliation moderated the association between maternal antipathy and VS activation to social reward anticipation, with significant associations found in participants with PTSD and HVs, but not in those with MDD and SSD. Results were not associated with general psychopathology or psychotropic medication use. Childhood maternal antipathy may confer risk for aberrant social reward anticipation in adulthood, and may thus be considered in interventions targeting reward expectations from social interactions.

Neural correlates of multisensory integration in the human brain: an ALE meta-analysis

Previous fMRI research identified superior temporal sulcus as central integration area for audiovisual stimuli. However, less is known about a general multisensory integration network across senses. Therefore, we conducted activation likelihood estimation meta-analysis with multiple sensory modalities to identify a common brain network. We included 49 studies covering all Aristotelian senses i.e., auditory, visual, tactile, gustatory, and olfactory stimuli. Analysis revealed significant activation in bilateral superior temporal gyrus, middle temporal gyrus, thalamus, right insula, and left inferior frontal gyrus. We assume these regions to be part of a general multisensory integration network comprising different functional roles. Here, thalamus operate as first subcortical relay projecting sensory information to higher cortical integration centers in superior temporal gyrus/sulcus while conflict-processing brain regions as insula and inferior frontal gyrus facilitate integration of incongruent information. We additionally performed meta-analytic connectivity modelling and found each brain region showed co-activations within the identified multisensory integration network. Therefore, by including multiple sensory modalities in our meta-analysis the results may provide evidence for a common brain network that supports different functional roles for multisensory integration.

Crossmodal benefits to vocal emotion perception in cochlear implant users

Speech comprehension counts as a benchmark outcome of cochlear implants (CIs)-disregarding the communicative importance of efficient integration of audiovisual (AV) socio-emotional information. We investigated effects of time-synchronized facial information on vocal emotion recognition (VER). In Experiment 1, 26 CI users and normal-hearing (NH) individuals classified emotions for auditory-only, AV congruent, or AV incongruent utterances. In Experiment 2, we compared crossmodal effects between groups with adaptive testing, calibrating auditory difficulty via voice morphs from emotional caricatures to anti-caricatures. CI users performed lower than NH individuals, and VER was correlated with life quality. Importantly, they showed larger benefits to VER with congruent facial emotional information even at equal auditory-only performance levels, suggesting that their larger crossmodal benefits result from deafness-related compensation rather than degraded acoustic representations. Crucially, vocal caricatures enhanced CI users' VER. Findings advocate AV stimuli during CI rehabilitation and suggest perspectives of caricaturing for both perceptual trainings and sound processor technology.

The contribution of the left precuneus to emotion memory in migraine without aura patients

Background

The impact of migraine without aura (MWoA) on cognitive function remains controversial, especially given the sparse literature on emotional memory.

Methods

Twenty seven MWoA patients and 25 healthy controls (HCs) were enrolled in this cross-sectional study. Emotional memory behavior was evaluated by combining incidental encoding with intentional encoding of five emotional categories of visual stimulus [positive valence + high arousal (PH), negative valence + high arousal (NH), positive valence + low arousal (PL), negative valence + low arousal (NL), and neutral (N)]. The recollection performance (Pr) was measured and compared. Then, the neural relevance was explored by correlating the Pr with gray matter volume (GMV) and resting-state functional connectivity (rs-FC) based on structural and functional magnetic resonance imaging.

Results

No significant differences in recollection performance or emotional enhancement of memory effect were observed. However, MWoA patients were more sensitive to the valence and arousal of emotional stimuli under incidental encoding. Significantly, the Pr-PH under incidental encoding and Pr-PL under intentional encoding were negatively correlated with the GMV of the left precuneus, and the rs-FC between the left precuneus and putamen was positively correlated with Pr-PL under intentional encoding in MWoA patients.

Conclusion

Our study demonstrated the tendency for the influence of migraine on emotional memory and revealed the left precuneus as a critical contributor to recollection performance, providing novel insights for understanding emotional memory and its neural mechanisms in MWoA patients.

The Role of the Interaction between the Inferior Parietal Lobule and Superior Temporal Gyrus in the Multisensory Go/No-go Task

Information from multiple sensory modalities interacts. Using functional magnetic resonance imaging (fMRI), we aimed to identify the neural structures correlated with how cooccurring sound modulates the visual motor response execution. The reaction time (RT) to audiovisual stimuli was significantly faster than the RT to visual stimuli. Signal detection analyses showed no significant difference in the perceptual sensitivity (d') between audiovisual and visual stimuli, while the response criteria (β or c) of the audiovisual stimuli was decreased compared to the visual stimuli. The functional connectivity between the left inferior parietal lobule (IPL) and bilateral superior temporal gyrus (STG) was enhanced in Go processing compared with No-go processing of audiovisual stimuli. Furthermore, the left precentral gyrus (PreCG) showed enhanced functional connectivity with the bilateral STG and other areas of the ventral stream in Go processing compared with No-go processing of audiovisual stimuli. These results revealed that the neuronal network correlated with modulations of the motor response execution after the presentation of both visual stimuli along with cooccurring sound in a multisensory Go/Nogo task, including the left IPL, left PreCG, bilateral STG and some areas of the ventral stream. The role of the interaction between the IPL and STG in transforming audiovisual information into motor behavior is discussed. The current study provides a new perspective for exploring potential brain mechanisms underlying how humans execute appropriate behaviors on the basis of multisensory information.

Cortical thickness abnormalities in patients with post-traumatic stress disorder: A vertex-based meta-analysis

Neuroimaging studies report altered cortical thickness in patients with post-traumatic stress disorder (PTSD), but the results are inconsistent. Using anisotropic effect-size seed-based d mapping (AES-SDM) software with its recently-developed meta-analytic thickness mask, we conducted a meta-analysis of published studies which used whole-brain surface-based morphometry, in order to define consistent cortical thickness alterations in PTSD patients. Eleven studies with 438 patients and 396 controls were included. Compared with all controls, patients with PTSD showed increased cortical thickness in right superior temporal gyrus, and in left and right superior frontal gyrus; the former survived in subgroup analysis of adult patients, and in subgroup comparison with only non-PTSD trauma-exposed controls, the latter in subgroup comparison with only non-trauma-exposed healthy controls. Cortical thickness in right superior frontal gyrus was positively associated with percentage of female patients, and cortical thickness in left superior frontal gyrus was positively associated with symptom severity measured by the clinician-administered PTSD scale. These robust results may help to elucidate the pathophysiology of PTSD.

Baseline global brain structural and functional alterations at the time of symptom onset can predict subsequent cognitive deterioration in drug-naïve first-episode schizophrenia patients: Evidence from a follow-up study

Alterations in the global brain gray matter volume (gGMV) and global functional connectivity density (gFCD) play a pivotal role in the cognitive impairment and further deterioration in schizophrenia. This study aimed to assess the correlation between alterations in the gGMV and gFCD at baseline (ΔgGMV and ΔgFCD), and the subsequent alterations of cognitive function in schizophrenia patients after 2-year antipsychotic treatment. Global-brain magnetic resonance imaging scans were acquired from 877 drug-naïve, first-episode schizophrenia patients at baseline and after two years of antipsychotic treatment with adequate dosage and duration, and 200 healthy controls. According to ΔgGMV at baseline, schizophrenia patients were divided into mild, moderate, and severe alteration groups. The MATRICS consensus cognitive battery and Global Deficit Score (GDS) were used to assess cognitive impairment. We found that ΔgGMV and ΔgFCD at baseline were significantly correlated with the severity of the cognitive deterioration (ΔGDS). The correlation coefficient indicated a significant positive correlation between baseline ΔgFCD and subsequent cognitive deterioration, with a relatively stronger relation in the mild alteration group (r = 0.31). In addition, there was a significant positive correlation between baseline ΔgGMV and subsequent cognitive deterioration, with a stronger relation in the moderate and severe alteration groups (r = 0.303; r = 0.302, respectively). Our results showed that ΔgGMV and ΔgFCD are correlated with the severity of cognitive deterioration after completion of a 2-year antipsychotic treatment in schizophrenia patients. These findings suggest that baseline alterations in gGMV and gFCD hold potential for predicting subsequent cognitive decline in schizophrenia.

Sound gaps boost emotional audiovisual integration independent of attention: Evidence from an ERP study

The emotion discrimination paradigm was adopted to study the effect of interrupted sound on visual emotional processing under different attentional states. There were two experiments: Experiment 1: judging facial expressions (explicit task), Experiment 2: judging the position of a bar (implicit task). In Experiment 1, ERP results showed that there was a sound gap accelerating the effect of P1 present only under neutral faces. In Experiment 2, the accelerating effect (P1) existed regardless of the emotional condition. Combining two experiments, P1 findings suggest that sound gap enhances bottom-up attention. The N170 and late positive component (LPC) were found to be regulated by emotion face in both experiments, with fear over the neutral. Comparing the two experiments, the explicit task induced a larger LPC than the implicit task. Overall, sound gaps boosted the audiovisual integration by bottom-up attention in early integration, while cognitive expectations led to top-down attention in late stages.

Cortical volume abnormalities in posttraumatic stress disorder: an ENIGMA-psychiatric genomics consortium PTSD workgroup mega-analysis

Studies of posttraumatic stress disorder (PTSD) report volume abnormalities in multiple regions of the cerebral cortex. However, findings for many regions, particularly regions outside commonly studied emotion-related prefrontal, insular, and limbic regions, are inconsistent and tentative. Also, few studies address the possibility that PTSD abnormalities may be confounded by comorbid depression. A mega-analysis investigating all cortical regions in a large sample of PTSD and control subjects can potentially provide new insight into these issues. Given this perspective, our group aggregated regional volumes data of 68 cortical regions across both hemispheres from 1379 PTSD patients to 2192 controls without PTSD after data were processed by 32 international laboratories using ENIGMA standardized procedures. We examined whether regional cortical volumes were different in PTSD vs. controls, were associated with posttraumatic stress symptom (PTSS) severity, or were affected by comorbid depression. Volumes of left and right lateral orbitofrontal gyri (LOFG), left superior temporal gyrus, and right insular, lingual and superior parietal gyri were significantly smaller, on average, in PTSD patients than controls (standardized coefficients = -0.111 to -0.068, FDR corrected P values < 0.039) and were significantly negatively correlated with PTSS severity. After adjusting for depression symptoms, the PTSD findings in left and right LOFG remained significant. These findings indicate that cortical volumes in PTSD patients are smaller in prefrontal regulatory regions, as well as in broader emotion and sensory processing cortical regions.

Weighted RSA: An Improved Framework on the Perception of Audio-visual Affective Speech in Left Insula and Superior Temporal Gyrus

Being able to accurately perceive the emotion expressed by the facial or verbal expression from others is critical to successful social interaction. However, only few studies examined the multimodal interactions on speech emotion, and there is no consistence in studies on the speech emotion perception. It remains unclear, how the speech emotion of different valence is perceived on the multimodal stimuli by our human brain. In this paper, we conducted a functional magnetic resonance imaging (fMRI) study with an event-related design, using dynamic facial expressions and emotional speech stimuli to express different emotions, in order to explore the perception mechanism of speech emotion in audio-visual modality. The representational similarity analysis (RSA), whole-brain searchlight analysis, and conjunction analysis of emotion were used to interpret the representation of speech emotion in different aspects. Significantly, a weighted RSA approach was creatively proposed to evaluate the contribution of each candidate model to the best fitted model and provided a supplement to RSA. The results of weighted RSA indicated that the fitted models were superior to all candidate models and the weights could be used to explain the representation of ROIs. The bilateral amygdala has been shown to be associated with the processing of both positive and negative emotions except neutral emotion. It is indicated that the left posterior insula and the left anterior superior temporal gyrus (STG) play important roles in the perception of multimodal speech emotion.

The adhesio interthalamica as a neuroanatomical marker of structural differences in healthy adult population

The adhesio interthalamica (AI) is a small midline brain structure that connects the left and right thalamus. According to in vivo data, between 2.3 and 22.3% of the general population lack the AI, and the question of whether this absence is more prevalent in males than in females is a matter of debate. Despite the existence of these demographic figures, it remains unclear how this distinctive feature affects healthy people, or what specific anatomic profile is related to the presence or absence of the AI. The aim of this study was to investigate whole-brain gray matter (GM) volumetric differences depending on the presence or absence of the AI. A total of 240 healthy adult volunteers completed one MRI scanning session. After the AI assessment, the data from 110 participants were included in the final sample, of which 12.9% of the participants (n = 31) presented complete AI absence vs. 32.9% of participants (n = 79) who presented complete AI presence. Then, whole-brain group comparison analysis revealed that the absent AI brain, compared to the present AI brain, was associated with lower GM volume in the premotor cortex, inferior frontal gyrus, and anterior temporal cortex. Interestingly, neuroscience research has linked emotional and cognitive control brain processing to the latter two regions. The importance of these findings lies in providing a neuroanatomical profile for the absent AI brain in healthy human adults.

Modality-general and modality-specific audiovisual valence processing

A fundamental question in affective neuroscience is whether there is a common hedonic system for valence processing independent of modality, or there are distinct neural systems for different modalities. To address this question, we used both region of interest and whole-brain representational similarity analyses on functional magnetic resonance imaging data to identify modality-general and modality-specific brain areas involved in valence processing across visual and auditory modalities. First, region of interest analyses showed that the superior temporal cortex was associated with both modality-general and auditory-specific models, while the primary visual cortex was associated with the visual-specific model. Second, the whole-brain searchlight analyses also identified both modality-general and modality-specific representations. The modality-general regions included the superior temporal, medial superior frontal, inferior frontal, precuneus, precentral, postcentral, supramarginal, paracentral lobule and middle cingulate cortices. The modality-specific regions included both perceptual cortices and higher-order brain areas. The valence representations derived from individualized behavioral valence ratings were consistent with these results. Together, these findings suggest both modality-general and modality-specific representations of valence.

Evoked and induced power oscillations linked to audiovisual integration of affect

Our affective experiences are influenced by combined multisensory information. Although the enhanced effects of congruent audiovisual information on our affective experiences have been well documented, the role of neural oscillations in the audiovisual integration of affective signals remains unclear. First, it is unclear whether oscillatory activity changes as a function of valence. Second, the function of phase-locked and non-phase-locked power changes in audiovisual integration of affect has not yet been clearly distinguished. To fill this gap, the present study performed time-frequency analyses on EEG data acquired while participants perceived positive, neutral and negative naturalistic video and music clips. A comparison between the congruent audiovisual condition and the sum of unimodal conditions was used to identify supra-additive (Audiovisual > Visual + Auditory) or sub-additive (Audiovisual < Visual + Auditory) integration effects. The results showed that early evoked sub-additive theta and sustained induced supra-additive delta and beta activities are linked to audiovisual integration of affect regardless of affective content.

EEG gamma activity to emotional movies in individuals with high traits of primary "successful" psychopathy

This study aimed to investigate emotional alterations in a community sample with primary psychopathic traits. Sixty males selected from a larger sample and divided in two groups, High (HP) and Low (LP) primary Psychopathy, watched 15 validated emotional movies divided in five categories (Erotic, Neutral, Fear, and the new Scenery and Compassion). Subjective responses and cortical activity in the EEG Gamma band (30-49 Hz) were recorded. Concerning self-reports, HP participants felt less anxious and anguished in response to Fear clips and less sad, touched and anguished by Compassion excerpts. Negative clips induced larger EEG Gamma activity in both groups compared to neutral and erotic movies, but Gamma activity to Fear movies was reduced in HP. This group showed also a small cortical response to positive Scenery movies, at the level of the Neutral ones. Source analysis showed in HP participants a reduced cortical activation to Fear in a large brain network, comprising the right prefrontal and temporal cortices and bilateral inferior parietal cortex. Results showed that primary psychopathy, in addition to the impaired response to Fear, was associated with a reduced response also to other specific categories of emotional stimuli, suggesting an altered affect on a broader scale.

An fMRI Study of Affective Congruence across Visual and Auditory Modalities

Evaluating multisensory emotional content is a part of normal day-to-day interactions. We used fMRI to examine brain areas sensitive to congruence of audiovisual valence and their overlap with areas sensitive to valence. Twenty-one participants watched audiovisual clips with either congruent or incongruent valence across visual and auditory modalities. We showed that affective congruence versus incongruence across visual and auditory modalities is identifiable on a trial-by-trial basis across participants. Representations of affective congruence were widely distributed with some overlap with the areas sensitive to valence. Regions of overlap included bilateral superior temporal cortex and right pregenual anterior cingulate. The overlap between the regions identified here and in the emotion congruence literature lends support to the idea that valence may be a key determinant of affective congruence processing across a variety of discrete emotions.