EEG oscillations reflect task effects for the change detection in vocal emotion

Electroencephalographic Abnormalities in a Patient Suffering from Long-Term Neuropsychological Complications following SARS-CoV-2 Infection

Introduction

Emotional apathy has recently been identified as a common symptom of long COVID. While recent meta-analyses have demonstrated generalized EEG slowing with the emergence of delta rhythms in patients hospitalized for severe SARS-CoV-2 infection, no EEG study or dopamine transporter scintigraphy (DaTSCAN) has been performed in patients with long COVID presenting with apathy. The objective of this case report was to explore the pathophysiology of neuropsychological symptoms in long COVID.

Case Presentation

A 47-year-old patient who developed a long COVID with prominent apathy following an initially clinically mild SARS-CoV-2 infection underwent neuropsychological assessment, cerebral MRI, DaTSCAN, and resting-state high-density EEG 7 months after SARS-CoV-2 infection. The EEG data were compared to those of 21 healthy participants. The patient presented with apathy, cognitive difficulties with dysexecutive syndrome, moderate attentional and verbal episodic memory disturbances, and resolution of premorbid mild gaming disorder, mild mood disturbances, and sleep disturbances. His MRI and DaTSCAN were unremarkable. EEG revealed a complex pattern of oscillatory abnormalities compared to the control group, with a strong increase in whole-scalp delta and beta band activity, as well as a decrease in alpha band activity. Overall, these effects were more prominent in the frontal-central-temporal region.

Conclusion

These results suggest widespread changes in EEG oscillatory patterns in a patient with long COVID characterized by neuropsychological complications with prominent apathy. Despite the inherent limitations of a case report, these results suggest dysfunction in the cortical networks involved in motivation and emotion.

Neurocognitive Dynamics of Prosodic Salience over Semantics during Explicit and Implicit Processing of Basic Emotions in Spoken Words

How language mediates emotional perception and experience is poorly understood. The present event-related potential (ERP) study examined the explicit and implicit processing of emotional speech to differentiate the relative influences of communication channel, emotion category and task type in the prosodic salience effect. Thirty participants (15 women) were presented with spoken words denoting happiness, sadness and neutrality in either the prosodic or semantic channel. They were asked to judge the emotional content (explicit task) and speakers' gender (implicit task) of the stimuli. Results indicated that emotional prosody (relative to semantics) triggered larger N100, P200 and N400 amplitudes with greater delta, theta and alpha inter-trial phase coherence (ITPC) and event-related spectral perturbation (ERSP) values in the corresponding early time windows, and continued to produce larger LPC amplitudes and faster responses during late stages of higher-order cognitive processing. The relative salience of prosodic and semantics was modulated by emotion and task, though such modulatory effects varied across different processing stages. The prosodic salience effect was reduced for sadness processing and in the implicit task during early auditory processing and decision-making but reduced for happiness processing in the explicit task during conscious emotion processing. Additionally, across-trial synchronization of delta, theta and alpha bands predicted the ERP components with higher ITPC and ERSP values significantly associated with stronger N100, P200, N400 and LPC enhancement. These findings reveal the neurocognitive dynamics of emotional speech processing with prosodic salience tied to stage-dependent emotion- and task-specific effects, which can reveal insights into understanding language and emotion processing from cross-linguistic/cultural and clinical perspectives.

Hierarchical scale convolutional neural network for facial expression recognition

Recognition of facial expressions plays an important role in understanding human behavior, classroom assessment, customer feedback, education, business, and many other human-machine interaction applications. Some researchers have realized that using features corresponding to different scales can improve the recognition accuracy, but there is a lack of a systematic study to utilize the scale information. In this work, we proposed a hierarchical scale convolutional neural network (HSNet) for facial expression recognition, which can systematically enhance the information extracted from the kernel, network, and knowledge scale. First, inspired by that the facial expression can be defined by different size facial action units and the power of sparsity, we proposed dilation Inception blocks to enhance kernel scale information extraction. Second, to supervise relatively shallow layers for learning more discriminated features from different size feature maps, we proposed a feature guided auxiliary learning approach to utilize high-level semantic features to guide the shallow layers learning. Last, since human cognitive ability can progressively be improved by learned knowledge, we mimicked such ability by knowledge transfer learning from related tasks. Extensive experiments on lab-controlled, synthesized, and in-the-wild databases showed that the proposed method substantially boosts performance, and achieved state-of-the-art accuracy on most databases. Ablation studies proved the effectiveness of modules in the proposed method.

EEG-based emotion recognition using 4D convolutional recurrent neural network

In this paper, we present a novel method, called four-dimensional convolutional recurrent neural network, which integrating frequency, spatial and temporal information of multichannel EEG signals explicitly to improve EEG-based emotion recognition accuracy. First, to maintain these three kinds of information of EEG, we transform the differential entropy features from different channels into 4D structures to train the deep model. Then, we introduce CRNN model, which is combined by convolutional neural network (CNN) and recurrent neural network with long short term memory (LSTM) cell. CNN is used to learn frequency and spatial information from each temporal slice of 4D inputs, and LSTM is used to extract temporal dependence from CNN outputs. The output of the last node of LSTM performs classification. Our model achieves state-of-the-art performance both on SEED and DEAP datasets under intra-subject splitting. The experimental results demonstrate the effectiveness of integrating frequency, spatial and temporal information of EEG for emotion recognition.

Subthalamic nucleus oscillations during vocal emotion processing are dependent of the motor asymmetry of Parkinson's disease

The subthalamic nucleus (STN) is involved in different aspects of emotional processes and more specifically in emotional prosody recognition. Recent studies on the behavioral effects of deep brain stimulation (DBS) in patients with Parkinson's disease (PD) have uncovered an asymmetry in vocal emotion decoding in PD, with left-onset PD patients showing deficits for the processing of happy voices. Whether and how PD asymmetry affects STN electrophysiological responses to emotional prosody, however, remains unknown. In the current study, local field potential activity was recorded from eight left- and six right-lateralized motor-onset PD patients (LOPD/ROPD) undergoing DBS electrodes implantation, while they listened to angry, happy and neutral voices. Time-frequency decomposition revealed that theta (2-6 Hz), alpha (6-12 Hz) and gamma (60-150 Hz) band responses to emotion were mostly bilateral with a differential pattern of response according to patient's sides-of onset. Conversely, beta-band (12-20 Hz and 20-30 Hz) emotional responses were mostly lateralized in the left STN for both patient groups. Furthermore, STN theta, alpha and gamma band responses to happiness were either absent (theta band) or reduced (alpha and gamma band) in the most affected STN hemisphere (contralateral to the side-of onset), while a late low-beta band left STN happiness-specific response was present in ROPD patients and did not occur in LOPD patients. Altogether, in this study, we demonstrate a complex pattern of oscillatory activity in the human STN in response to emotional voices and reveal a crucial influence of disease laterality on STN low-frequency oscillatory activity.

Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller

OBJECTIVE

Prior studies of emotional cognition have found that emotion-based bimodal face and voice stimuli can elicit larger event-related potential (ERP) amplitudes and enhance neural responses compared with visual-only emotional face stimuli. Recent studies on brain-computer interface have shown that emotional face stimuli have significantly improved the performance of the traditional P300 speller system, but its performance needs to be further improved for practical applications. Therefore, we herein propose a novel audiovisual P300 speller based on bimodal emotional cognition to further improve the performance of the P300 system.

METHODS

The audiovisual P300 speller we proposed is based on happy emotions, with visual and auditory stimuli that consist of several pairs of smiling faces and audible chuckles (E-AV spelling paradigm) of different ages and sexes. The control paradigm was the visual-only emotional face P300 speller (E-V spelling paradigm).

RESULTS

We compared the ERP amplitudes, accuracy, and raw bit rate between the E-AV and E-V spelling paradigms. The target stimuli elicited significantly increased P300 amplitudes (p < .05) and P600 amplitudes (p < .05) in the E-AV spelling paradigm compared with those in the E-V paradigm. The E-AV spelling paradigm also significantly improved the spelling accuracy and the raw bit rate compared with those in the E-V paradigm at one superposition (p < .05) and at two superpositions (p < .05).

SIGNIFICANCE

The proposed emotion-based audiovisual spelling paradigm not only significantly improves the performance of the P300 speller, but also provides a basis for the development of various bimodal P300 speller systems, which is a step forward in the clinical application of brain-computer interfaces.

Influence of attention on bimodal integration during emotional change decoding: ERP evidence

Recent findings on audiovisual emotional interactions suggest that selective attention affects cross-sensory interaction from an early processing stage. However, the influence of attention manipulation on facial-vocal integration during emotional change perception is still elusive at this point. To address this issue, we asked participants to detect emotional changes conveyed by prosodies (vocal task) or facial expressions (facial task) while facial, vocal, and facial-vocal expressions were presented. At the same time, behavioral responses and electroencephalogram (EEG) were recorded. Behavioral results showed that bimodal emotional changes were detected with shorter response latencies compared to each unimodal condition, suggesting that bimodal emotional cues facilitated emotional change detection. Moreover, while the P3 amplitudes were larger for the bimodal change condition than for the sum of the two unimodal conditions regardless of attention direction, the N1 amplitudes were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions under the attend-voice condition, but not under the attend-face condition. These findings suggest that selective attention modulates facial-vocal integration during emotional change perception in early sensory processing, but not in late cognitive processing stages.

The Functional Role of Neural Oscillations in Non-Verbal Emotional Communication

Effective interpersonal communication depends on the ability to perceive and interpret nonverbal emotional expressions from multiple sensory modalities. Current theoretical models propose that visual and auditory emotion perception involves a network of brain regions including the primary sensory cortices, the superior temporal sulcus (STS), and orbitofrontal cortex (OFC). However, relatively little is known about how the dynamic interplay between these regions gives rise to the perception of emotions. In recent years, there has been increasing recognition of the importance of neural oscillations in mediating neural communication within and between functional neural networks. Here we review studies investigating changes in oscillatory activity during the perception of visual, auditory, and audiovisual emotional expressions, and aim to characterize the functional role of neural oscillations in nonverbal emotion perception. Findings from the reviewed literature suggest that theta band oscillations most consistently differentiate between emotional and neutral expressions. While early theta synchronization appears to reflect the initial encoding of emotionally salient sensory information, later fronto-central theta synchronization may reflect the further integration of sensory information with internal representations. Additionally, gamma synchronization reflects facilitated sensory binding of emotional expressions within regions such as the OFC, STS, and, potentially, the amygdala. However, the evidence is more ambiguous when it comes to the role of oscillations within the alpha and beta frequencies, which vary as a function of modality (or modalities), presence or absence of predictive information, and attentional or task demands. Thus, the synchronization of neural oscillations within specific frequency bands mediates the rapid detection, integration, and evaluation of emotional expressions. Moreover, the functional coupling of oscillatory activity across multiples frequency bands supports a predictive coding model of multisensory emotion perception in which emotional facial and body expressions facilitate the processing of emotional vocalizations.

EEG Oscillation Evidences of Enhanced Susceptibility to Emotional Stimuli during Adolescence

Background: Our recent event-related potential (ERP) study showed that adolescents are more emotionally sensitive to negative events compared to adults, regardless of the valence strength of the events. The current work aimed to confirm this age-related difference in response to emotional stimuli of diverse intensities by examining Electroencephalography (EEG) oscillatory power in time-frequency analysis.

Methods: Time-frequency analyses were performed on the EEG data recorded for highly negative (HN), moderately negative (MN) and Neutral pictures in 20 adolescents and 20 adults during a covert emotional task. The results showed a significant age by emotion interaction effect in the theta and beta oscillatory power during the 500–600 ms post stimulus.

Results: Adolescents showed significantly less pronounced theta synchronization (ERS, 5.5–7.5 Hz) for HN stimuli, and larger beta desynchronization (ERD; 18–20 Hz) for both HN and MN stimuli, in comparison with neutral stimuli. By contrast, adults exhibited no significant emotion effects in theta and beta frequency bands. In addition, the analysis of the alpha spectral power (10.5–12 Hz; 850–950 ms) showed a main effect of emotion, while the emotion by age interaction was not significant. Irrespective of adolescents or adults, HN and MN stimuli elicited enhanced alpha suppression compared to Neutral stimuli, while the alpha power was similar across HN and MN conditions.

Conclusions: These results confirmed prior findings that adolescents are more sensitive to emotionally negative stimuli compared to adults, regardless of emotion intensity, possibly due to the developing prefrontal control system during adolescence.

The integration of facial and vocal cues during emotional change perception: EEG markers

The ability to detect emotional changes is of primary importance for social living. Though emotional signals are often conveyed by multiple modalities, how emotional changes in vocal and facial modalities integrate into a unified percept has yet to be directly investigated. To address this issue, we asked participants to detect emotional changes delivered by facial, vocal and facial-vocal expressions while behavioral responses and electroencephalogram were recorded. Behavioral results showed that bimodal emotional changes were detected with higher accuracy and shorter response latencies compared with each unimodal condition. Moreover, the detection of emotional change, regardless of modalities, was associated with enhanced amplitudes in the N2 and P3 component, as well as greater theta synchronization. More importantly, the P3 amplitudes and theta synchronization were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions. The superadditive responses in P3 amplitudes and theta synchronization were both positively correlated with the magnitude of the bimodal superadditivity in accuracy. These behavioral and electrophysiological data consistently illustrated an effect of audiovisual integration during the detection of emotional changes, which is most likely mediated by the P3 activity and theta oscillations in brain responses.