Towards a cross-modal perspective of emotional perception in social anxiety: review and future directions

Unattended Emotional Prosody Affects Visual Processing of Facial Expressions in Mandarin-Speaking Chinese: A Comparison With English-Speaking Canadians

Emotional cues from different modalities have to be integrated during communication, a process that can be shaped by an individual’s cultural background. We explored this issue in 25 Chinese participants by examining how listening to emotional prosody in Mandarin influenced participants’ gazes at emotional faces in a modified visual search task. We also conducted a cross-cultural comparison between data of this study and that of our previous work in English-speaking Canadians using analogous methodology. In both studies, eye movements were recorded as participants scanned an array of four faces portraying fear, anger, happy, and neutral expressions, while passively listening to a pseudo-utterance expressing one of the four emotions (Mandarin utterance in this study; English utterance in our previous study). The frequency and duration of fixations to each face were analyzed during 5 seconds after the onset of faces, both during the presence of the speech (early time window) and after the utterance ended (late time window). During the late window, Chinese participants looked more frequently and longer at faces conveying congruent emotions as the speech, consistent with findings from English-speaking Canadians. Cross-cultural comparison further showed that Chinese, but not Canadians, looked more frequently and longer at angry faces, which may signal potential conflicts and social threats. We hypothesize that the socio-cultural norms related to harmony maintenance in the Eastern culture promoted Chinese participants’ heightened sensitivity to, and deeper processing of, angry cues, highlighting culture-specific patterns in how individuals scan their social environment during emotion processing.

The Neural Correlates of Face-Voice-Integration in Social Anxiety Disorder

Faces and voices are very important sources of threat in social anxiety disorder (SAD), a common psychiatric disorder where core elements are fears of social exclusion and negative evaluation. Previous research in social anxiety evidenced increased cerebral responses to negative facial or vocal expressions and also generally increased hemodynamic responses to voices and faces. But it is unclear if also the cerebral process of face-voice-integration is altered in SAD. Applying functional magnetic resonance imaging, we investigated the correlates of the audiovisual integration of dynamic faces and voices in SAD as compared to healthy individuals. In the bilateral midsections of the superior temporal sulcus (STS) increased integration effects in SAD were observed driven by greater activation increases during audiovisual stimulation as compared to auditory stimulation. This effect was accompanied by increased functional connectivity with the visual association cortex and a more anterior position of the individual integration maxima along the STS in SAD. These findings demonstrate that the audiovisual integration of facial and vocal cues in SAD is not only systematically altered with regard to intensity and connectivity but also the individual location of the integration areas within the STS. These combined findings offer a novel perspective on the neuronal representation of social signal processing in individuals suffering from SAD.

An investigation of emotion recognition training to reduce symptoms of social anxiety in adolescence

This study aimed to examine the effect of emotion recognition training on social anxiety symptoms among adolescents, aged 15-18 years. The study included a screening session, which identified participants who scored above a cut-off on a self-report measure of social anxiety for enrolment into a randomized controlled trial (Clinical Trials ID: NCT02550379). Participants were randomized to an intervention condition designed to increase the perception of happiness over disgust in ambiguous facial expressions or a sham intervention control condition, and completed self-report measures of social anxiety, fear of negative evaluation, anxiety-related disorders, and depressive symptoms. The intervention group demonstrated a strong shift in the balance point at which they perceived happiness over disgust in ambiguous facial expressions. This increase in positive perception was not associated with any changes in the primary outcome of social anxiety; however, some evidence of improvement in symptomatology was observed on one of a number of secondary outcomes. Those in the intervention group had lower depression symptoms at 2-week follow-up, compared to those in the control group who received the sham intervention training. Potential reasons for why the shift in balance point measurement was not associated with a concurrent shift in symptoms of social anxiety are discussed.

Facial and prosodic emotion recognition in social anxiety disorder

INTRODUCTION

Patients with social anxiety disorder (SAD) have a cognitive preference to negatively evaluate emotional information. In particular, the preferential biases in prosodic emotion recognition in SAD have been much less explored. The present study aims to investigate whether SAD patients retain negative evaluation biases across visual and auditory modalities when given sufficient response time to recognise emotions.

METHODS

Thirty-one SAD patients and 31 age- and gender-matched healthy participants completed a culturally suitable non-verbal emotion recognition task and received clinical assessments for social anxiety and depressive symptoms. A repeated measures analysis of variance was conducted to examine group differences in emotion recognition.

RESULTS

Compared to healthy participants, SAD patients were significantly less accurate at recognising facial and prosodic emotions, and spent more time on emotion recognition. The differences were mainly driven by the lower accuracy and longer reaction times for recognising fearful emotions in SAD patients. Within the SAD patients, lower accuracy of sad face recognition was associated with higher severity of depressive and social anxiety symptoms, particularly with avoidance symptoms.

CONCLUSION

These findings may represent a cross-modality pattern of avoidance in the later stage of identifying negative emotions in SAD. This pattern may be linked to clinical symptom severity.

Involuntary processing of social dominance cues from bimodal face-voice displays

Social-rank cues communicate social status or social power within and between groups. Information about social-rank is fluently processed in both visual and auditory modalities. So far, the investigation on the processing of social-rank cues has been limited to studies in which information from a single modality was assessed or manipulated. Yet, in everyday communication, multiple information channels are used to express and understand social-rank. We sought to examine the (in)voluntary nature of processing of facial and vocal signals of social-rank using a cross-modal Stroop task. In two experiments, participants were presented with face-voice pairs that were either congruent or incongruent in social-rank (i.e. social dominance). Participants' task was to label face social dominance while ignoring the voice, or label voice social dominance while ignoring the face. In both experiments, we found that face-voice incongruent stimuli were processed more slowly and less accurately than were the congruent stimuli in the face-attend and the voice-attend tasks, exhibiting classical Stroop-like effects. These findings are consistent with the functioning of a social-rank bio-behavioural system which consistently and automatically monitors one's social standing in relation to others and uses that information to guide behaviour.

Selective attention to emotional prosody in social anxiety: a dichotic listening study

The majority of evidence on social anxiety (SA)-linked attentional biases to threat comes from research using facial expressions. Emotions are, however, communicated through other channels, such as voice. Despite its importance in the interpretation of social cues, emotional prosody processing in SA has been barely explored. This study investigated whether SA is associated with enhanced processing of task-irrelevant angry prosody. Fifty-three participants with high and low SA performed a dichotic listening task in which pairs of male/female voices were presented, one to each ear, with either the same or different prosody (neutral or angry). Participants were instructed to focus on either the left or right ear and to identify the speaker's gender in the attended side. Our main results show that, once attended, task-irrelevant angry prosody elicits greater interference than does neutral prosody. Surprisingly, high socially anxious participants were less prone to distraction from attended-angry (compared to attended-neutral) prosody than were low socially anxious individuals. These findings emphasise the importance of examining SA-related biases across modalities.

A Preliminary Study on the Biased Attention and Interpretation in the Recognition of Face-Body Compound of the Individuals with Social Anxiety

The present study aimed to investigate the processes through which individuals with social anxiety (SA) attend to and interpret compound emotional expressions of the face and body. Incongruent face-body compound images that combined an angry face (or body) with a fearful, sad, or happy body (or face) were presented to a SA group (n = 22) and a healthy control (HC) group (n = 22). The participants were instructed to interpret the emotional state of the image, and their eye-movements and behavioral responses were measured. The results revealed that both group showed longer scanpath length during the recognition of compound images which combined angry face with angry, fearful, sadness, or happy body. The SA group also showed longer scanpath length in congruent face-body compound images of fear and sad. Additionally, the SA group fixated for a shorter period of time on the face and longer on the body than the HC group. Regarding emotion interpretation, the SA group was more likely to interpret the emotional state of incongruent face-body compound images based on the body than the HC group. These findings provide a preliminary observation that individuals with SA showed different attentional bias pattern by congruency of face-body compound images and that it might have biased their interpretations of the emotional states.

Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains

In everyday life, multiple sensory channels jointly trigger emotional experiences and one channel may alter processing in another channel. For example, seeing an emotional facial expression and hearing the voice’s emotional tone will jointly create the emotional experience. This example, where auditory and visual input is related to social communication, has gained considerable attention by researchers. However, interactions of visual and auditory emotional information are not limited to social communication but can extend to much broader contexts including human, animal, and environmental cues. In this article, we review current research on audiovisual emotion processing beyond face-voice stimuli to develop a broader perspective on multimodal interactions in emotion processing. We argue that current concepts of multimodality should be extended in considering an ecologically valid variety of stimuli in audiovisual emotion processing. Therefore, we provide an overview of studies in which emotional sounds and interactions with complex pictures of scenes were investigated. In addition to behavioral studies, we focus on neuroimaging, electro- and peripher-physiological findings. Furthermore, we integrate these findings and identify similarities or differences. We conclude with suggestions for future research.