Prediction of subjective ratings of emotional pictures by EEG features

Higher cortical excitability to negative emotions involved in musculoskeletal pain in Parkinson's disease

OBJECTIVE

Changes in brain structure and neurotransmitter systems are involved in pain in Parkinson's disease (PD), and emotional factors are closely related to pain. Our study applied electroencephalography (EEG) to investigate the role of emotion in PD patients with chronic musculoskeletal pain.

METHODS

Forty-two PD patients with chronic musculoskeletal pain and 38 without were enrolled. EEG data were recorded under resting conditions, and while viewing pictures with neutral, positive, and negative content. We compared spectrum power, functional connectivity, and late positive potential (LPP), an event-related potential (ERP), between the groups.

RESULTS

PD patients with pain tended to have higher scores for the Hamilton Rating Scale for Depression (HRSD). In the resting EEG, mean β-band amplitude was significantly higher in patients with pain than in those without. Logistic regression analysis showed that higher HRSD scores and higher mean β-band amplitude were associated with pain. ERP analysis revealed that the amplitudes of LPP difference waves (the absolute difference between positive and negative condition LPP and neutral condition LPP) at the central-parietal region were significantly reduced in patients with pain (P = 0.029). Spearman correlation analysis showed that the amplitudes of late (700-1000 ms) negative versus neutral condition LPP difference waves were negatively correlated with pain intensity, assessed by visual analogue scale, (r = -0.393, P = 0.010) and HRSD scores (r = -0.366, P = 0.017).

CONCLUSION

Dopaminergic and non-dopaminergic systems may be involved in musculoskeletal pain in PD by increasing β-band activity and weakening the connection of the θ-band at the central-parietal region. PD patients with musculoskeletal pain have higher cortical excitability to negative emotions. The changes in pain-related EEG may be used as electrophysiological markers and therapeutic targets in PD patients with chronic pain.

Image-Evoked Emotion Recognition for Hearing-Impaired Subjects with EEG Signals

In recent years, there has been a growing interest in the study of emotion recognition through electroencephalogram (EEG) signals. One particular group of interest are individuals with hearing impairments, who may have a bias towards certain types of information when communicating with those in their environment. To address this, our study collected EEG signals from both hearing-impaired and non-hearing-impaired subjects while they viewed pictures of emotional faces for emotion recognition. Four kinds of feature matrices, symmetry difference, and symmetry quotient based on original signal and differential entropy (DE) were constructed, respectively, to extract the spatial domain information. The multi-axis self-attention classification model was proposed, which consists of local attention and global attention, combining the attention model with convolution through a novel architectural element for feature classification. Three-classification (positive, neutral, negative) and five-classification (happy, neutral, sad, angry, fearful) tasks of emotion recognition were carried out. The experimental results show that the proposed method is superior to the original feature method, and the multi-feature fusion achieved a good effect in both hearing-impaired and non-hearing-impaired subjects. The average classification accuracy for hearing-impaired subjects and non-hearing-impaired subjects was 70.2% (three-classification) and 50.15% (five-classification), and 72.05% (three-classification) and 51.53% (five-classification), respectively. In addition, by exploring the brain topography of different emotions, we found that the discriminative brain regions of the hearing-impaired subjects were also distributed in the parietal lobe, unlike those of the non-hearing-impaired subjects.

A pediatric near-infrared spectroscopy brain-computer interface based on the detection of emotional valence

Brain-computer interfaces (BCIs) are being investigated as an access pathway to communication for individuals with physical disabilities, as the technology obviates the need for voluntary motor control. However, to date, minimal research has investigated the use of BCIs for children. Traditional BCI communication paradigms may be suboptimal given that children with physical disabilities may face delays in cognitive development and acquisition of literacy skills. Instead, in this study we explored emotional state as an alternative access pathway to communication. We developed a pediatric BCI to identify positive and negative emotional states from changes in hemodynamic activity of the prefrontal cortex (PFC). To train and test the BCI, 10 neurotypical children aged 8–14 underwent a series of emotion-induction trials over four experimental sessions (one offline, three online) while their brain activity was measured with functional near-infrared spectroscopy (fNIRS). Visual neurofeedback was used to assist participants in regulating their emotional states and modulating their hemodynamic activity in response to the affective stimuli. Child-specific linear discriminant classifiers were trained on cumulatively available data from previous sessions and adaptively updated throughout each session. Average online valence classification exceeded chance across participants by the last two online sessions (with 7 and 8 of the 10 participants performing better than chance, respectively, in Sessions 3 and 4). There was a small significant positive correlation with online BCI performance and age, suggesting older participants were more successful at regulating their emotional state and/or brain activity. Variability was seen across participants in regards to BCI performance, hemodynamic response, and discriminatory features and channels. Retrospective offline analyses yielded accuracies comparable to those reported in adult affective BCI studies using fNIRS. Affective fNIRS-BCIs appear to be feasible for school-aged children, but to further gauge the practical potential of this type of BCI, replication with more training sessions, larger sample sizes, and end-users with disabilities is necessary.

Deep Convolutional Neural Network-Based Visual Stimuli Classification Using Electroencephalography Signals of Healthy and Alzheimer’s Disease Subjects

Visual perception is an important part of human life. In the context of facial recognition, it allows us to distinguish between emotions and important facial features that distinguish one person from another. However, subjects suffering from memory loss face significant facial processing problems. If the perception of facial features is affected by memory impairment, then it is possible to classify visual stimuli using brain activity data from the visual processing regions of the brain. This study differentiates the aspects of familiarity and emotion by the inversion effect of the face and uses convolutional neural network (CNN) models (EEGNet, EEGNet SSVEP (steady-state visual evoked potentials), and DeepConvNet) to learn discriminative features from raw electroencephalography (EEG) signals. Due to the limited number of available EEG data samples, Generative Adversarial Networks (GAN) and Variational Autoencoders (VAE) are introduced to generate synthetic EEG signals. The generated data are used to pretrain the models, and the learned weights are initialized to train them on the real EEG data. We investigate minor facial characteristics in brain signals and the ability of deep CNN models to learn them. The effect of face inversion was studied, and it was observed that the N170 component has a considerable and sustained delay. As a result, emotional and familiarity stimuli were divided into two categories based on the posture of the face. The categories of upright and inverted stimuli have the smallest incidences of confusion. The model’s ability to learn the face-inversion effect is demonstrated once more.

Brain response to color stimuli: an EEG study with nonlinear approach

Color perception is a major guiding factor in the evolutionary process of human civilization, but most of the neurological background of the same are yet unknown. This work attempts to address this area with an EEG based neuro-cognitive study on response of brain to different color stimuli. With respect to a Grey baseline seven colors of the VIBGYOR were shown to 16 participants with normal color vision and corresponding EEG signals from different lobes (Frontal, Occipital & Parietal) were recorded. In an attempt to quantify the brain response while watching these colors, the corresponding EEG signals were analysed using two of the latest state of the art non-linear techniques (MFDFA and MFDXA) of dealing complex time series. MFDFA revealed that for all the participants the spectral width, and hence the complexity of the EEG signals, reaches a maximum while viewing color Blue, followed by colors Red and Green in all the brain lobes. MFDXA, on the other hand, suggests a lower degree of inter and intra lobe correlation while watching the VIBGYOR colors compared to baseline Grey, hinting towards a post processing of visual information. We hope that along with the novelty of methodologies, the unique outcomes of this study may leave a long term impact in the domain of color perception research.

Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels

Background/Introduction

Emotion detection classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle compound and ambiguous labels. We explore the feasibility of using crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We hypothesize that training with labels that are representative of the diversity of human interpretation of an image will result in predictions that are similarly representative on a disjoint test set. We also hypothesize that crowdsourcing can generate distributions which mirror those generated in a lab setting.

Methods

We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we used Microworkers to acquire labels for 207 CAFE images. We evaluate both unfiltered workers as well as workers selected through a short crowd filtration process. We then train two versions of a ResNet-152 neural network on soft-target CAFE labels using the original 100 annotations provided with the dataset: (1) a classifier trained with traditional one-hot encoded labels, and (2) a classifier trained with vector labels representing the distribution of CAFE annotator responses. We compare the resulting softmax output distributions of the two classifiers with a 2-sample independent t-test of L1 distances between the classifier's output probability distribution and the distribution of human labels.

Results

While agreement with CAFE is weak for unfiltered crowd workers, the filtered crowd agree with the CAFE labels 100% of the time for happy, neutral, sad and "fear + surprise", and 88.8% for "anger + disgust". While the F1-score for a one-hot encoded classifier is much higher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the output probability vector of the crowd-trained classifier more closely resembles the distribution of human labels (t=3.2827, p=0.0014).

Conclusions

For many applications of affective computing, reporting an emotion probability distribution that accounts for the subjectivity of human interpretation can be more useful than an absolute label. Crowdsourcing, including a sufficient filtering mechanism for selecting reliable crowd workers, is a feasible solution for acquiring soft-target labels.

Evaluation of Emotional Satisfaction Using Questionnaires in Voice-Based Human–AI Interaction

With the development of artificial intelligence technology, voice-based intelligent systems (VISs), such as AI speakers and virtual assistants, are intervening in human life. VISs are emerging in a new way, called human–AI interaction, which is different from existing human–computer interaction. Using the Kansei engineering approach, we propose a method to evaluate user satisfaction during interaction between a VIS and a user-centered intelligent system. As a user satisfaction evaluation method, a VIS comprising four types of design parameters was developed. A total of 23 subjects were considered for interaction with the VIS, and user satisfaction was measured using Kansei words (KWs). The questionnaire scores collected through KWs were analyzed using exploratory factor analysis. ANOVA was used to analyze differences in emotion. On the “pleasurability” and “reliability” axes, it was confirmed that among the four design parameters, “sentence structure of the answer” and “number of trials to get the right answer for a question” affect the emotional satisfaction of users. Four satisfaction groups were derived according to the level of the design parameters. This study can be used as a reference for conducting an integrated emotional satisfaction assessment using emotional metrics such as biosignals and facial expressions.

Data augmentation for enhancing EEG-based emotion recognition with deep generative models

OBJECTIVE

The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models.

APPROACH

Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation ways, full and partial usage strategies. For the full usage strategy, all of the generated data are augmented to the training dataset without judging the quality of the generated data, while for the partial usage, only high-quality data are selected and appended to the training dataset. These three methods are called conditional Wasserstein GAN (cWGAN), selective VAE (sVAE), and selective WGAN (sWGAN).

MAIN RESULTS

To evaluate the effectiveness of these proposed methods, we perform a systematic experimental study on two public EEG datasets for emotion recognition, namely, SEED and DEAP. We first generate realistic-like EEG training data in two forms: power spectral density and differential entropy. Then, we augment the original training datasets with a different number of generated realistic-like EEG data. Finally, we train support vector machines and deep neural networks with shortcut layers to build affective models using the original and augmented training datasets. The experimental results demonstrate that our proposed data augmentation methods based on generative models outperform the existing data augmentation approaches such as conditional VAE, Gaussian noise, and rotational data augmentation. We also observe that the number of generated data should be less than 10 times of the original training dataset to achieve the best performance.

SIGNIFICANCE

The augmented training datasets produced by our proposed sWGAN method significantly enhance the performance of EEG-based emotion recognition models.

Independent Components of EEG Activity Correlating with Emotional State

Among brain-computer interface studies, electroencephalography (EEG)-based emotion recognition is receiving attention and some studies have performed regression analyses to recognize small-scale emotional changes; however, effective brain regions in emotion regression analyses have not been identified yet. Accordingly, this study sought to identify neural activities correlating with emotional states in the source space. We employed independent component analysis, followed by a source localization method, to obtain distinct neural activities from EEG signals. After the identification of seven independent component (IC) clusters in a k-means clustering analysis, group-level regression analyses using frequency band power of the ICs were performed based on Russell's valence-arousal model. As a result, in the regression of the valence level, an IC cluster located in the cuneus predicted both high- and low-valence states and two other IC clusters located in the left precentral gyrus and the precuneus predicted the low-valence state. In the regression of the arousal level, the IC cluster located in the cuneus predicted both high- and low-arousal states and two posterior IC clusters located in the cingulate gyrus and the precuneus predicted the high-arousal state. In this proof-of-concept study, we revealed neural activities correlating with specific emotional states across participants, despite individual differences in emotional processing.

Effect of brightness of visual stimuli on EEG signals

The aim of this study was to examine brightness effect, which is the perceptual property of visual stimuli, on brain responses obtained during visual processing of these stimuli. For this purpose, brain responses of the brain to changes in brightness were explored comparatively using different emotional images (pleasant, unpleasant and neutral) with different luminance levels. In the study, electroencephalography recordings from 12 different electrode sites of 31 healthy participants were used. The power spectra obtained from the analysis of the recordings using short time Fourier transform were analyzed, and a statistical analysis was performed on features extracted from these power spectra. Statistical findings were compared with those obtained from behavioral data. The results showed that the brightness of visual stimuli affected the power of brain responses depending on frequency, time and location. According to the statistically verified findings, the increase in the brightness of pleasant and neutral images increased the average power of responses in the parietal and occipital regions whereas the increase in the brightness of unpleasant images decreased the average power of responses in these regions. Moreover, the statistical results obtained for unpleasant images were found to be in accordance with the behavioral data. The results revealed that the brightness of visual stimuli could be represented by changing the activity power of the brain cortex. The findings emphasized that the brightness of visual stimuli should be viewed as an important parameter in studies using emotional image techniques such as image classification, emotion evaluation and neuro-marketing.

Percept-related EEG classification using machine learning approach and features of functional brain connectivity

Machine learning is a promising approach for electroencephalographic (EEG) trials classification. Its efficiency is largely determined by the feature extraction and selection techniques reducing the dimensionality of input data. Dimensionality reduction is usually implemented via the mathematical approaches (e.g., principal component analysis, linear discriminant analysis, etc.) regardless of the origin of analyzed data. We hypothesize that since EEG features are determined by certain neurophysiological processes, they should have distinctive characteristics in spatiotemporal domain. If so, it is possible to specify the set of EEG principal features based on the prior knowledge about underlying neurophysiological processes. To test this hypothesis, we consider the classification of EEG trials related to the perception of ambiguous visual stimuli. We observe that EEG features, underlying the different ambiguous stimuli interpretations, are defined by the network properties of neuronal activity. Having analyzed functional neural interactions, we specify the brain area in which neural network architecture exhibits differences for different classes of EEG trials. We optimize the feedforward multilayer perceptron and develop a strategy for the training set selection to maximize the classification accuracy, being 85% when all channels are used. The revealed localization of the percept-related features allows about 95% accuracy, when the number of channels is reduced up to 90%. Obtained results can be used for classification of EEG trials associated with more complex cognitive tasks. Taking into account that cognitive activity is subserved by a distributed functional cortical network, its topological properties have to be considered when selecting optimal features for EEG trial classification.

Decoding electroencephalographic signals for direction in brain-computer interface using echo state network and Gaussian readouts

BACKGROUND

Noninvasive brain-computer interfaces (BCI) for movement control via an electroencephalogram (EEG) have been extensively investigated. However, most previous studies decoded user intention for movement directions based on sensorimotor rhythms during motor imagery. BCI systems based on mapping imagery movement of body parts (e.g., left or right hands) to movement directions (left or right directional movement of a machine or cursor) are less intuitive and less convenient due to the complex training procedures. Thus, direct decoding methods for detecting user intention about movement directions are urgently needed.

METHODS

Here, we describe a novel direct decoding method for user intention about the movement directions using the echo state network and Gaussian readouts. Importantly parameters in the network were optimized using the genetic algorithm method to achieve better decoding performance. We tested the decoding performance of this method with four healthy subjects and an inexpensive wireless EEG system containing 14 channels and then compared the performance outcome with that of a conventional machine learning method.

RESULTS

We showed that this decoding method successfully classified eight directions of intended movement (approximately 95% of an accuracy).

CONCLUSIONS

We suggest that the echo state network and Gaussian readouts can be a useful decoding method to directly read user intention of movement directions even using an inexpensive and portable EEG system.

The Approach Behavior to Angry Words in Athletes-A Pilot Study

An increasing number of studies have found that athletes have a higher level of aggression than non-athletes. Anger is an important factor in the generation of aggressive behavior, and anger has been found to relate to both approach behavior and avoidance behavior. The present pilot study compared the aggression level of athletes and non-athletes using the Buss-Perry Aggression Questionnaire, and examined the responses of participants to anger-related stimuli using the manikin task, a paradigm that measures approach-avoidance behavior. In total, 15 athletes and 15 non-athletes finished the questionnaire and the manikin task, which included two conditions. In the anger approach condition, participants were asked to approach anger-associated words and to avoid neutral words. The instructions for the anger avoidance condition were the opposite (i.e., move away from the anger-associated words and toward the neutral words). Brain activity was recorded during the manikin task. Results showed that, compared with non-athletes, athletes had significantly higher physical aggression on the questionnaire. The athlete group showed significantly shorter reaction times in anger approach condition than anger avoidance condition. Theta oscillation activity induced during the anger approach condition was significantly lower than that during the anger avoidance condition in the athlete group. No significant correlation was found in present pilot study. These findings may suggest that when anger-related stimuli are present, athletes are more likely to approach, indicating stronger behavioral approach motivation that may result in aggressive behavior.

Frontal EEG Asymmetry and Middle Line Power Difference in Discrete Emotions

A traditional model of emotion cannot explain the differences in brain activities between two discrete emotions that are similar in the valence-arousal coordinate space. The current study elicited two positive emotions (amusement and tenderness) and two negative emotions (anger and fear) that are similar in both valence and arousal dimensions to examine the differences in brain activities in these emotional states. Frontal electroencephalographic (EEG) asymmetry and midline power in three bands (theta, alpha and beta) were measured when participants watched affective film excerpts. Significant differences were detected between tenderness and amusement on FP1/FP2 theta asymmetry, F3/F4 theta and alpha asymmetry. Significant differences between anger and fear on FP1/FP2 theta asymmetry and F3/F4 alpha asymmetry were also observed. For midline power, midline theta power could distinguish two negative emotions, while midline alpha and beta power could effectively differentiate two positive emotions. Liking and dominance were also related to EEG features. Stepwise multiple linear regression results revealed that frontal alpha and theta asymmetry could predict the subjective feelings of two positive and two negative emotions in different patterns. The binary classification accuracy, which used EEG frontal asymmetry and midline power as features and support vector machine (SVM) as classifiers, was as high as 64.52% for tenderness and amusement and 78.79% for anger and fear. The classification accuracy was improved after adding these features to other features extracted across the scalp. These findings indicate that frontal EEG asymmetry and midline power might have the potential to recognize discrete emotions that are similar in the valence-arousal coordinate space.

Classifying the Perceptual Interpretations of a Bistable Image Using EEG and Artificial Neural Networks

In order to classify different human brain states related to visual perception of ambiguous images, we use an artificial neural network (ANN) to analyze multichannel EEG. The classifier built on the basis of a multilayer perceptron achieves up to 95% accuracy in classifying EEG patterns corresponding to two different interpretations of the Necker cube. The important feature of our classifier is that trained on one subject it can be used for the classification of EEG traces of other subjects. This result suggests the existence of common features in the EEG structure associated with distinct interpretations of bistable objects. We firmly believe that the significance of our results is not limited to visual perception of the Necker cube images; the proposed experimental approach and developed computational technique based on ANN can also be applied to study and classify different brain states using neurophysiological data recordings. This may give new directions for future research in the field of cognitive and pathological brain activity, and for the development of brain-computer interfaces.

Rethinking self-reported measure in subjective evaluation of assistive technology

Self-reporting is used as a subjective measure of usability study of technology solutions. In assistive technology research, more than often the ‘a coordinator’ directly assist the ‘subject’ in the scoring process. This makes the rating process slower and also introduces bias, such as, ‘Forer effect’ and/or ‘Hawthorne’ effect. To address these issues we propose to use technology mediated interaction between the ‘subject’ and ‘the coordinator’ in evaluating assistive technology solutions. The goal is to combine both the qualitative and quantitative scores to create a relatively unbiased rating system. Empirical studies were performed on two different datasets in order to illustrate the utility of the proposed approach. It was observed that, the proposed hybrid rating is relatively unbiased for usability study.

Therapeutic Applications of BCI Technologies

Brain-computer interface (BCI) technology can restore communication and control to people who are severely paralyzed. There has been speculation that this technology might also be useful for a variety of diverse therapeutic applications. This survey considers possible ways that BCI technology can be applied to motor rehabilitation following stroke, Parkinson's disease, and psychiatric disorders. We consider potential neural signals as well as the design and goals of BCI-based therapeutic applications. These diverse applications all share a reliance on neuroimaging and signal processing technologies. At the same time, each of these potential applications presents a series of unique challenges.