Data augmentation for enhancing EEG-based emotion recognition with deep generative models

Synthesizing affective neurophysiological signals using generative models: A review paper

The integration of emotional intelligence in machines is an important step in advancing human-computer interaction. This demands the development of reliable end-to-end emotion recognition systems. However, the scarcity of public affective datasets presents a challenge. In this literature review, we emphasize the use of generative models to address this issue in neurophysiological signals, particularly Electroencephalogram (EEG) and Functional Near-Infrared Spectroscopy (fNIRS). We provide a comprehensive analysis of different generative models used in the field, examining their input formulation, deployment strategies, and methodologies for evaluating the quality of synthesized data. This review serves as a comprehensive overview, offering insights into the advantages, challenges, and promising future directions in the application of generative models in emotion recognition systems. Through this review, we aim to facilitate the progression of neurophysiological data augmentation, thereby supporting the development of more efficient and reliable emotion recognition systems.

EEG-based emotion recognition systems; comprehensive study

Emotion recognition technology through EEG signal analysis is currently a fundamental concept in artificial intelligence. This recognition has major practical implications in emotional health care, human-computer interaction, and so on. This paper provides a comprehensive study of different methods for extracting electroencephalography (EEG) features for emotion recognition from four different perspectives, including time domain features, frequency domain features, time-frequency features, and nonlinear features. We summarize the current pattern recognition methods adopted in most related works, and with the rapid development of deep learning (DL) attracting the attention of researchers in this field, we pay more attention to deep learning-based studies and analyse the characteristics, advantages, disadvantages, and applicable scenarios. Finally, the current challenges and future development directions in this field were summarized. This paper can help novice researchers in this field gain a systematic understanding of the current status of emotion recognition research based on EEG signals and provide ideas for subsequent related research.

On the role of generative artificial intelligence in the development of brain-computer interfaces

Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.

Research on Fatigue Driving Detection Technology Based on CA-ACGAN

Driver fatigue represents a significant peril to global traffic safety, necessitating the advancement of potent fatigue monitoring methodologies to bolster road safety. This research introduces a conditional generative adversarial network with a classification head that integrates convolutional and attention mechanisms (CA-ACGAN) designed for the precise identification of fatigue driving states through the analysis of electroencephalography (EEG) signals. First, this study constructed a 4D feature data model capable of mirroring drivers' fatigue state, meticulously analyzing the EEG signals' frequency, spatial, and temporal dimensions. Following this, we present the CA-ACGAN framework, a novel integration of attention schemes, the bottleneck residual block, and the Transformer element. This integration was designed to refine the processing of EEG signals significantly. In utilizing a conditional generative adversarial network equipped with a classification header, the framework aims to distinguish fatigue states effectively. Moreover, it addresses the scarcity of authentic data through the generation of superior-quality synthetic data. Empirical outcomes illustrate that the CA-ACGAN model surpasses various extant methods in the fatigue detection endeavor on the SEED-VIG public dataset. Moreover, juxtaposed with leading-edge GAN models, our model exhibits an efficacy in in producing high-quality data that is clearly superior. This investigation confirms the CA-ACGAN model's utility in fatigue driving identification and suggests fresh perspectives for deep learning applications in time series data generation and processing.

Cross-subject EEG-based emotion recognition through dynamic optimization of random forest with sparrow search algorithm

The objective of EEG-based emotion recognition is to classify emotions by decoding signals, with potential applications in the fields of artificial intelligence and bioinformatics. Cross-subject emotion recognition is more difficult than intra-subject emotion recognition. The poor adaptability of classification model parameters is a significant factor of low accuracy in cross-subject emotion recognition. We propose a model of a dynamically optimized Random Forest based on the Sparrow Search Algorithm (SSA-RF). The decision trees number (DTN) and the leave minimum number (LMN) of the RF are dynamically optimized by the SSA. 12 features are used to construct feature combinations for selecting the optimal feature combination. DEAP and SEED datasets are employed for testing the performance of SSA-RF. The experimental results show that the accuracy of binary classification is 76.81% on DEAP, and the accuracy of triple classification is 75.96% on SEED based on SSA-RF, which are both higher than that of traditional RF. This study provides new insights for the development of cross-subject emotion recognition, and has significant theoretical value.

Mini review: Challenges in EEG emotion recognition

Electroencephalography (EEG) stands as a pioneering tool at the intersection of neuroscience and technology, offering unprecedented insights into human emotions. Through this comprehensive review, we explore the challenges and opportunities associated with EEG-based emotion recognition. While recent literature suggests promising high accuracy rates, these claims necessitate critical scrutiny for their authenticity and applicability. The article highlights the significant challenges in generalizing findings from a multitude of EEG devices and data sources, as well as the difficulties in data collection. Furthermore, the disparity between controlled laboratory settings and genuine emotional experiences presents a paradox within the paradigm of emotion research. We advocate for a balanced approach, emphasizing the importance of critical evaluation, methodological standardization, and acknowledging the dynamism of emotions for a more holistic understanding of the human emotional landscape.

Deep Generative Models: The winning key for large and easily accessible ECG datasets?

Large high-quality datasets are essential for building powerful artificial intelligence (AI) algorithms capable of supporting advancement in cardiac clinical research. However, researchers working with electrocardiogram (ECG) signals struggle to get access and/or to build one. The aim of the present work is to shed light on a potential solution to address the lack of large and easily accessible ECG datasets. Firstly, the main causes of such a lack are identified and examined. Afterward, the potentials and limitations of cardiac data generation via deep generative models (DGMs) are deeply analyzed. These very promising algorithms have been found capable not only of generating large quantities of ECG signals but also of supporting data anonymization processes, to simplify data sharing while respecting patients' privacy. Their application could help research progress and cooperation in the name of open science. However several aspects, such as a standardized synthetic data quality evaluation and algorithm stability, need to be further explored.

Generation of synthetic EEG data for training algorithms supporting the diagnosis of major depressive disorder

Introduction

Major depressive disorder (MDD) is the most common mental disorder worldwide, leading to impairment in quality and independence of life. Electroencephalography (EEG) biomarkers processed with machine learning (ML) algorithms have been explored for objective diagnoses with promising results. However, the generalizability of those models, a prerequisite for clinical application, is restricted by small datasets. One approach to train ML models with good generalizability is complementing the original with synthetic data produced by generative algorithms. Another advantage of synthetic data is the possibility of publishing the data for other researchers without risking patient data privacy. Synthetic EEG time-series have not yet been generated for two clinical populations like MDD patients and healthy controls.

Methods

We first reviewed 27 studies presenting EEG data augmentation with generative algorithms for classification tasks, like diagnosis, for the possibilities and shortcomings of recent methods. The subsequent empirical study generated EEG time-series based on two public datasets with 30/28 and 24/29 subjects (MDD/controls). To obtain baseline diagnostic accuracies, convolutional neural networks (CNN) were trained with time-series from each dataset. The data were synthesized with generative adversarial networks (GAN) consisting of CNNs. We evaluated the synthetic data qualitatively and quantitatively and finally used it for re-training the diagnostic model.

Results

The reviewed studies improved their classification accuracies by between 1 and 40% with the synthetic data. Our own diagnostic accuracy improved up to 10% for one dataset but not significantly for the other. We found a rich repertoire of generative models in the reviewed literature, solving various technical issues. A major shortcoming in the field is the lack of meaningful evaluation metrics for synthetic data. The few studies analyzing the data in the frequency domain, including our own, show that only some features can be produced truthfully.

Discussion

The systematic review combined with our own investigation provides an overview of the available methods for generating EEG data for a classification task, their possibilities, and shortcomings. The approach is promising and the technical basis is set. For a broad application of these techniques in neuroscience research or clinical application, the methods need fine-tuning facilitated by domain expertise in (clinical) EEG research.

Image2Brain: a cross-modality model for blind stereoscopic image quality ranking

Objective.Human beings perceive stereoscopic image quality through the cerebral visual cortex, which is a complex brain activity. As a solution, the quality of stereoscopic images can be evaluated more accurately by attempting to replicate the human perception from electroencephalogram (EEG) signals on image quality in a machine, which is different from previous stereoscopic image quality assessment methods focused only on the extraction of image features.Approach.Our proposed method is based on a novel image-to-brain (I2B) cross-modality model including a spatial-temporal EEG encoder (STEE) and an I2B deep convolutional generative adversarial network (I2B-DCGAN). Specifically, the EEG representations are first learned by STEE as real samples of I2B-DCGAN, which is designed to extract both quality and semantic features from the stereoscopic images by a semantic-guided image encoder, and utilize a generator to conditionally create the corresponding EEG features for images. Finally, the generated EEG features are classified to predict the image perceptual quality level.Main results.Extensive experimental results on the collected brain-visual multimodal stereoscopic image quality ranking database, demonstrate that the proposed I2B cross-modality model can better emulate the visual perception mechanism of the human brain and outperform the other methods by achieving an average accuracy of 95.95%.Significance.The proposed method can convert the learned stereoscopic image features into brain representations without EEG signals during testing. Further experiments verify that the proposed method has good generalization ability on new datasets and the potential for practical applications.

Decoding auditory-evoked response in affective states using wearable around-ear EEG system

Objective.In this paper, an around-ear EEG system is investigated as an alternative methodology to conventional scalp-EEG-based systems in classifying human affective states in the arousal-valence domain evoked in response to auditory stimuli.Approach.EEG recorded from around the ears is compared to EEG collected according to the international 10-20 system in terms of efficacy in an affective state classification task. A wearable device with eight dry EEG channels is designed for ear-EEG acquisition in this study. Twenty-one subjects participated in an experiment consisting of six sessions over three days using both ear and scalp-EEG acquisition methods. Experimental tasks consisted of listening to an auditory stimulus and self-reporting the elicited emotion in response to the said stimulus. Various features were used in tandem with asymmetry methods to evaluate binary classification performances of arousal and valence states using ear-EEG signals in comparison to scalp-EEG.Main results.We achieve an average accuracy of 67.09% ± 6.14 for arousal and 66.61% ± 6.14 for valence after training a multi-layer extreme learning machine with ear-EEG signals in a subject-dependent context in comparison to scalp-EEG approach which achieves an average accuracy of 68.59% ± 6.26 for arousal and 67.10% ± 4.99 for valence. In a subject-independent context, the ear-EEG approach achieves 63.74% ± 3.84 for arousal and 64.32% ± 6.38 for valence while the scalp-EEG approach achieves 64.67% ± 6.91 for arousal and 64.86% ± 5.95 for valence. The best results show no significant differences between ear-EEG and scalp-EEG signals for classifications of affective states.Significance.To the best of our knowledge, this paper is the first work to explore the use of around-ear EEG signals in emotion monitoring. Our results demonstrate the potential use of around-ear EEG systems for the development of emotional monitoring setups that are more suitable for use in daily affective life log systems compared to conventional scalp-EEG setups.

Review of Performance Improvement of a Noninvasive Brain-computer Interface in Communication and Motor Control for Clinical Applications

Brain-computer interfaces (BCI) enable direct communication between the brain and a computer or other external devices. They can extend a person's degree of freedom by either strengthening or substituting the human peripheral working capacity. Moreover, their potential clinical applications in medical fields include rehabilitation, affective computing, communication, and control. Over the last decade, noninvasive BCI systems such as electroencephalogram (EEG) have progressed from simple statistical models to deep learning models, with performance improvement over time and enhanced computational power. However, numerous challenges pertaining to the clinical use of BCI systems remain, e.g., the lack of sufficient data to learn more possible features for robust and reliable classification. However, compared with fields such as computer vision and speech recognition, the training samples in the medical BCI field are limited as they target patients who face difficulty generating EEG data compared with healthy control. Because deep learning models incorporate several parameters, they require considerably more data than other conventional methods. Thus, deep learning models have not been thoroughly leveraged in medical BCI. This study summarizes the state-of-the-art progress of the BCI system over the last decade, highlighting critical challenges and solutions.

Optimizing 1D-CNN-Based Emotion Recognition Process through Channel and Feature Selection from EEG Signals

EEG-based emotion recognition has numerous real-world applications in fields such as affective computing, human-computer interaction, and mental health monitoring. This offers the potential for developing IOT-based, emotion-aware systems and personalized interventions using real-time EEG data. This study focused on unique EEG channel selection and feature selection methods to remove unnecessary data from high-quality features. This helped improve the overall efficiency of a deep learning model in terms of memory, time, and accuracy. Moreover, this work utilized a lightweight deep learning method, specifically one-dimensional convolutional neural networks (1D-CNN), to analyze EEG signals and classify emotional states. By capturing intricate patterns and relationships within the data, the 1D-CNN model accurately distinguished between emotional states (HV/LV and HA/LA). Moreover, an efficient method for data augmentation was used to increase the sample size and observe the performance deep learning model using additional data. The study conducted EEG-based emotion recognition tests on SEED, DEAP, and MAHNOB-HCI datasets. Consequently, this approach achieved mean accuracies of 97.6, 95.3, and 89.0 on MAHNOB-HCI, SEED, and DEAP datasets, respectively. The results have demonstrated significant potential for the implementation of a cost-effective IoT device to collect EEG signals, thereby enhancing the feasibility and applicability of the data.

Subject-independent EEG classification based on a hybrid neural network

A brain-computer interface (BCI) based on the electroencephalograph (EEG) signal is a novel technology that provides a direct pathway between human brain and outside world. For a traditional subject-dependent BCI system, a calibration procedure is required to collect sufficient data to build a subject-specific adaptation model, which can be a huge challenge for stroke patients. In contrast, subject-independent BCI which can shorten or even eliminate the pre-calibration is more time-saving and meets the requirements of new users for quick access to the BCI. In this paper, we design a novel fusion neural network EEG classification framework that uses a specially designed generative adversarial network (GAN), called a filter bank GAN (FBGAN), to acquire high-quality EEG data for augmentation and a proposed discriminative feature network for motor imagery (MI) task recognition. Specifically, multiple sub-bands of MI EEG are first filtered using a filter bank approach, then sparse common spatial pattern (CSP) features are extracted from multiple bands of filtered EEG data, which constrains the GAN to maintain more spatial features of the EEG signal, and finally we design a convolutional recurrent network classification method with discriminative features (CRNN-DF) to recognize MI tasks based on the idea of feature enhancement. The hybrid neural network proposed in this study achieves an average classification accuracy of 72.74 ± 10.44% (mean ± std) in four-class tasks of BCI IV-2a, which is 4.77% higher than the state-of-the-art subject-independent classification method. A promising approach is provided to facilitate the practical application of BCI.

Diagnosis of autism spectrum disorder based on functional brain networks and machine learning

Autism is a multifaceted neurodevelopmental condition whose accurate diagnosis may be challenging because the associated symptoms and severity vary considerably. The wrong diagnosis can affect families and the educational system, raising the risk of depression, eating disorders, and self-harm. Recently, many works have proposed new methods for the diagnosis of autism based on machine learning and brain data. However, these works focus on only one pairwise statistical metric, ignoring the brain network organization. In this paper, we propose a method for the automatic diagnosis of autism based on functional brain imaging data recorded from 500 subjects, where 242 present autism spectrum disorder considering the regions of interest throughout Bootstrap Analysis of Stable Cluster map. Our method can distinguish the control group from autism spectrum disorder patients with high accuracy. Indeed the best performance provides an AUC near 1.0, which is higher than that found in the literature. We verify that the left ventral posterior cingulate cortex region is less connected to an area in the cerebellum of patients with this neurodevelopment disorder, which agrees with previous studies. The functional brain networks of autism spectrum disorder patients show more segregation, less distribution of information across the network, and less connectivity compared to the control cases. Our workflow provides medical interpretability and can be used on other fMRI and EEG data, including small data sets.

Generative adversarial networks in EEG analysis: an overview

Electroencephalogram (EEG) signals have been utilized in a variety of medical as well as engineering applications. However, one of the challenges associated with recording EEG data is the difficulty of recording large amounts of data. Consequently, data augmentation is a potential solution to overcome this challenge in which the objective is to increase the amount of data. Inspired by the success of Generative Adversarial Networks (GANs) in image processing applications, generating artificial EEG data from the limited recorded data using GANs has seen recent success. This article provides an overview of various techniques and approaches of GANs for augmenting EEG signals. We focus on the utility of GANs in different applications including Brain-Computer Interface (BCI) paradigms such as motor imagery and P300-based systems, in addition to emotion recognition, epileptic seizures detection and prediction, and various other applications. We address in this article how GANs have been used in each study, the impact of using GANs on the model performance, the limitations of each algorithm, and future possibilities for developing new algorithms. We emphasize the utility of GANs in augmenting the limited EEG data typically available in the studied applications.

Data Augmentation Effects on Highly Imbalanced EEG Datasets for Automatic Detection of Photoparoxysmal Responses

Photosensitivity is a neurological disorder in which a person's brain produces epileptic discharges, known as Photoparoxysmal Responses (PPRs), when it receives certain visual stimuli. The current standardized diagnosis process used in hospitals consists of submitting the subject to the Intermittent Photic Stimulation process and attempting to trigger these phenomena. The brain activity is measured by an Electroencephalogram (EEG), and the clinical specialists manually look for the PPRs that were provoked during the session. Due to the nature of this disorder, long EEG recordings may contain very few PPR segments, meaning that a highly imbalanced dataset is available. To tackle this problem, this research focused on applying Data Augmentation (DA) to create synthetic PPR segments from the real ones, improving the balance of the dataset and, thus, the global performance of the Machine Learning techniques applied for automatic PPR detection. K-Nearest Neighbors and a One-Hidden-Dense-Layer Neural Network were employed to evaluate the performance of this DA stage. The results showed that DA is able to improve the models, making them more robust and more able to generalize. A comparison with the results obtained from a previous experiment also showed a performance improvement of around 20% for the Accuracy and Specificity measurements without Sensitivity suffering any losses. This project is currently being carried out with subjects at Burgos University Hospital, Spain.

Classification of Motor Imagery EEG Signals Based on Data Augmentation and Convolutional Neural Networks

In brain-computer interface (BCI) systems, motor imagery electroencephalography (MI-EEG) signals are commonly used to detect participant intent. Many factors, including low signal-to-noise ratios and few high-quality samples, make MI classification difficult. In order for BCI systems to function, MI-EEG signals must be studied. In pattern recognition and other fields, deep learning approaches have recently been successfully applied. In contrast, few effective deep learning algorithms have been applied to BCI systems, especially MI-based systems. In this paper, we address these problems from two aspects based on the characteristics of EEG signals: first, we proposed a combined time-frequency domain data enhancement method. This method guarantees that the size of the training data is effectively increased while maintaining the intrinsic composition of the data. Second, our design consists of a parallel CNN that takes both raw EEG images and images transformed through continuous wavelet transform (CWT) as inputs. We conducted classification experiments on a public data set to verify the effectiveness of the algorithm. According to experimental results based on the BCI Competition IV Dataset2a, the average classification accuracy is 97.61%. A comparison of the proposed algorithm with other algorithms shows that it performs better in classification. The algorithm can be used to improve the classification performance of MI-based BCIs and BCI systems created for people with disabilities.

A Multifaceted benchmarking of synthetic electronic health record generation models

Synthetic health data have the potential to mitigate privacy concerns in supporting biomedical research and healthcare applications. Modern approaches for data generation continue to evolve and demonstrate remarkable potential. Yet there is a lack of a systematic assessment framework to benchmark methods as they emerge and determine which methods are most appropriate for which use cases. In this work, we introduce a systematic benchmarking framework to appraise key characteristics with respect to utility and privacy metrics. We apply the framework to evaluate synthetic data generation methods for electronic health records data from two large academic medical centers with respect to several use cases. The results illustrate that there is a utility-privacy tradeoff for sharing synthetic health data and further indicate that no method is unequivocally the best on all criteria in each use case, which makes it evident why synthetic data generation methods need to be assessed in context.

E2SGAN: EEG-to-SEEG translation with generative adversarial networks

High-quality brain signal data recorded by Stereoelectroencephalography (SEEG) electrodes provide clinicians with clear guidance for presurgical assessments for epilepsy surgeries. SEEG, however, is limited to selected patients with epilepsy due to its invasive procedure. In this work, a brain signal synthesis framework is presented to synthesize SEEG signals from non-invasive EEG signals. First, a strategy to determine the matching relation between EEG and SEEG channels is presented by considering both signal correlation and spatial distance. Second, the EEG-to-SEEG generative adversarial network (E2SGAN) is proposed to precisely synthesize SEEG data from the simultaneous EEG data. Although the widely adopted magnitude spectra has proved to be informative in EEG tasks, it leaves much to be desired in the setting of signal synthesis. To this end, instantaneous frequency spectra is introduced to further represent the alignment of the signal. Correlative spectral attention (CSA) is proposed to enhance the discriminator of E2SGAN by capturing the correlation between each pair of EEG and SEEG frequencies. The weighted patch prediction (WPP) technique is devised to ensure robust temporal results. Comparison experiments on real-patient data demonstrate that E2SGAN outperforms baseline methods in both temporal and frequency domains. The perturbation experiment reveals that the synthesized results have the potential to capture abnormal discharges in epileptic patients before seizures.

Multimodal Emotion Classification Method and Analysis of Brain Functional Connectivity Networks

Since multimodal emotion classification in different human states has rarely been studied, this paper explores the emotional mechanisms of the brain functional connectivity networks after emotional stimulation. We devise a multimodal emotion classification method fusing a brain functional connectivity network based on electroencephalography (EEG) and eye gaze (ECFCEG) to study emotional mechanisms. First, the nonlinear phase lag index (PLI) and phase-locked value (PLV) are calculated to construct the multiband brain functional connectivity networks, which are then converted into binary brain networks, and the seven features of the binary brain networks are extracted. At the same time, the features of the eye gaze signals are extracted. Then, a fusion algorithm called kernel canonical correlation analysis, based on feature level and randomization (FRKCCA), is executed for feature-level fusion (FLF) of brain functional connectivity networks and eye gaze. Finally, support vector machines (SVMs) are utilized to classify positive and negative emotions in multiple frequency bands with single modal features and multimodal features. The experimental results demonstrate that multimodal complementary representation properties can effectively improve the accuracy of emotion classification, achieving a classification accuracy of 91.32±1.81%. The classification accuracy of pupil diameter in the valence dimension is higher than that of additional features. In addition, the average emotion classification effect of the valence dimension is preferable to that of arousal. Our findings demonstrate that the brain functional connectivity networks of the right brain exhibit a deficiency. In particular, the information processing ability of the right temporal (RT) and right posterior (RP) regions is weak in the low frequency after emotional stimulation; Conversely, phase synchronization of the brain functional connectivity networks based on PLI is stronger than that of PLV.

Representation Learning and Pattern Recognition in Cognitive Biometrics: A Survey

Cognitive biometrics is an emerging branch of biometric technology. Recent research has demonstrated great potential for using cognitive biometrics in versatile applications, including biometric recognition and cognitive and emotional state recognition. There is a major need to summarize the latest developments in this field. Existing surveys have mainly focused on a small subset of cognitive biometric modalities, such as EEG and ECG. This article provides a comprehensive review of cognitive biometrics, covering all the major biosignal modalities and applications. A taxonomy is designed to structure the corresponding knowledge and guide the survey from signal acquisition and pre-processing to representation learning and pattern recognition. We provide a unified view of the methodological advances in these four aspects across various biosignals and applications, facilitating interdisciplinary research and knowledge transfer across fields. Furthermore, this article discusses open research directions in cognitive biometrics and proposes future prospects for developing reliable and secure cognitive biometric systems.

An EEG-based systematic explainable detection framework for probing and localizing abnormal patterns in Alzheimer's disease

OBJECTIVE

Electroencephalography (EEG) is a potential source of downstream biomarkers for the early diagnosis of Alzheimer's disease (AD) due to its low-cost, non-invasive, and portable advantages. Accurately detecting AD-induced patterns from EEG signals is essential for understanding AD-related neurodegeneration at the EEG level and further evaluating the risk of AD at an early stage. This paper proposes a deep learning-based, functional explanatory framework that probes AD abnormalities from short-sequence EEG data.

APPROACH

The framework is a learning-based automatic detection system consisting of three encoding pathways that analyze EEG signals in frequency, complexity, and synchronous domains. We integrated the proposed EEG descriptors with the neural network components into one learning system to detect AD patterns. A transfer learning-based model was used to learn the deep representations, and a modified generative adversarial module was attached to the model to overcome feature sparsity. Furthermore, we utilized activation mapping to obtain the AD-related neurodegeneration at brain rhythm, dynamic complexity, and functional connectivity levels.

MAIN RESULTS

The proposed framework can accurately (100%) detect AD patterns based on our raw EEG recordings without delicate preprocessing. Meanwhile, the system indicates that 1) the power of different brain rhythms exhibits abnormal in the frontal lobes of AD patients, and such abnormality spreads to central lobes in the alpha and beta rhythms, 2) the difference in nonlinear complexity varies with the temporal scales, and 3) all the connections of pair-wise brain regions except bilateral temporal connectivity are weak in AD patterns. The proposed method outperforms other related methods in detection performance.

SIGNIFICANCE

We provide a new method for revealing abnormalities and corresponding localizations in different feature domains of EEG from AD patients. This study is a significant foundation for our future work on identifying individuals at high risk of AD at an early stage.

EEG Feature Extraction and Data Augmentation in Emotion Recognition

Emotion recognition is a challenging problem in Brain-Computer Interaction (BCI). Electroencephalogram (EEG) gives unique information about brain activities that are created due to emotional stimuli. This is one of the most substantial advantages of brain signals in comparison to facial expression, tone of voice, or speech in emotion recognition tasks. However, the lack of EEG data and high dimensional EEG recordings lead to difficulties in building effective classifiers with high accuracy. In this study, data augmentation and feature extraction techniques are proposed to solve the lack of data problem and high dimensionality of data, respectively. In this study, the proposed method is based on deep generative models and a data augmentation strategy called Conditional Wasserstein GAN (CWGAN), which is applied to the extracted features to regenerate additional EEG features. DEAP dataset is used to evaluate the effectiveness of the proposed method. Finally, a standard support vector machine and a deep neural network with different tunes were implemented to build effective models. Experimental results show that using the additional augmented data enhances the performance of EEG-based emotion recognition models. Furthermore, the mean accuracy of classification after data augmentation is increased 6.5% for valence and 3.0% for arousal, respectively.

Semi-supervised generative and discriminative adversarial learning for motor imagery-based brain-computer interface

Convolutional neural networks (CNNs), which can recognize structural/configuration patterns in data with different architectures, have been studied for feature extraction. However, challenges remain regarding leveraging advanced deep learning methods in BCIs. We focus on problems of small-sized training samples and interpretability of the learned parameters and leverages a semi-supervised generative and discriminative learning framework that effectively utilizes synthesized samples with real samples to discover class-discriminative features. Our framework learns the distributional characteristics of EEG signals in an embedding space using a generative model. By using artificially generated and real EEG signals, our framework finds class-discriminative spatio-temporal feature representations that help to correctly discriminate input EEG signals. It is noteworthy that the framework facilitates the exploitation of real, unlabeled samples to better uncover the underlying patterns inherent in a user’s EEG signals. To validate our framework, we conducted experiments comparing our method with conventional linear models by utilizing variants of three existing CNN architectures as generator networks and measuring the performance on three public datasets. Our framework exhibited statistically significant improvements over the competing methods. We investigated the learned network via activation pattern maps and visualized generated artificial samples to empirically justify the stability and neurophysiological plausibility of our model.

Identifying similarities and differences in emotion recognition with EEG and eye movements among Chinese, German, and French people

OBJECTIVE

Cultures have essential influences on emotions. However, most studies on cultural influences on emotions are in the areas of psychology and neuroscience, while the existing affective models are mostly built with data from the same culture. In this paper, we identify the similarities and differences among Chinese, German, and French individuals in emotion recognition with electroencephalogram (EEG) and eye movements from an affective computing perspective.

APPROACH

Three experimental settings were designed: intraculture subject dependent, intraculture subject independent, and cross-culture subject independent. EEG and eye movements are acquired simultaneously from Chinese, German, and French subjects while watching positive, neutral, and negative movie clips. The affective models for Chinese, German, and French subjects are constructed by using machine learning algorithms. A systematic analysis is performed from four aspects: affective model performance, neural patterns, complementary information from different modalities, and cross-cultural emotion recognition.

MAIN RESULTS

From emotion recognition accuracies, we find that EEG and eye movements can adapt to Chinese, German, and French cultural diversities and that a cultural in-group advantage phenomenon does exist in emotion recognition with EEG. From the topomaps of EEG, we find that the gamma and beta bands exhibit decreasing activities for Chinese, while for German and French, theta and alpha bands exhibit increasing activities. From confusion matrices and attentional weights, we find that EEG and eye movements have complementary characteristics. From a cross-cultural emotion recognition perspective, we observe that German and French people share more similarities in topographical patterns and attentional weight distributions than Chinese people while the data from Chinese are a good fit for test data but not suitable for training data for the other two cultures.

SIGNIFICANCE

Our experimental results provide concrete evidence of the in-group advantage phenomenon, cultural influences on emotion recognition, and different neural patterns among Chinese, German, and French individuals.

Esophageal optical coherence tomography image synthesis using an adversarially learned variational autoencoder

Endoscopic optical coherence tomography (OCT) imaging offers a non-invasive way to detect esophageal lesions on the microscopic scale, which is of clinical potential in the early diagnosis and treatment of esophageal cancers. Recent studies focused on applying deep learning-based methods in esophageal OCT image analysis and achieved promising results, which require a large data size. However, traditional data augmentation techniques generate samples that are highly correlated and sometimes far from reality, which may not lead to a satisfied trained model. In this paper, we proposed an adversarial learned variational autoencoder (AL-VAE) to generate high-quality esophageal OCT samples. The AL-VAE combines the generative adversarial network (GAN) and variational autoencoder (VAE) in a simple yet effective way, which preserves the advantages of VAEs, such as stable training and nice latent manifold, and requires no extra discriminators. Experimental results verified the proposed method achieved better image quality in generating esophageal OCT images when compared with the state-of-the-art image synthesis network, and its potential in improving deep learning model performance was also evaluated by esophagus segmentation.

Implementation of a Convolutional Neural Network for Eye Blink Artifacts Removal From the Electroencephalography Signal

Electroencephalography (EEG) signals are disrupted by technical and physiological artifacts. One of the most common artifacts is the natural activity that results from the movement of the eyes and the blinking of the subject. Eye blink artifacts (EB) spread across the entire head surface and make EEG signal analysis difficult. Methods for the elimination of electrooculography (EOG) artifacts, such as independent component analysis (ICA) and regression, are known. The aim of this article was to implement the convolutional neural network (CNN) to eliminate eye blink artifacts. To train the CNN, a method for augmenting EEG signals was proposed. The results obtained from the CNN were compared with the results of the ICA and regression methods for the generated and real EEG signals. The results obtained indicate a much better performance of the CNN in the task of removing eye-blink artifacts, in particular for the electrodes located in the central part of the head.

Decoding motor imagery tasks using ESI and hybrid feature CNN

Objective. BCI based on motor imaging electroencephalogram (MI-EEG) can be useful in a natural interaction system. In this paper, a new framework is proposed to solve the MI-EEG binary classification problem. Approach. Electrophysiological source imaging (ESI) technology is used to solve the influence of volume conduction effect and improve spatial resolution. Continuous wavelet transform (CWT) and best time of interest (TOI) are combined to extract the optimal discriminant spatial-frequency features. Finally, a CNN network with seven convolution layers is used to classify the features. In addition, we also validated several new data augment methods to solve the problem of small data sets and reduce network over-fitting. Main results. The model achieved an average classification accuracy of 93.2% and 95.4% on the BCI Competition III IVa and high-gamma data sets, which is better than most of the published advanced algorithms. By selecting the best TOI for each subject, the classification accuracy rate increased by about 2%. The effects of 4 data augment methods on the classification results were also verified. Among them, the noise addition and overlap methods are better than the other two, and the classification accuracy is improved by at least 4%. On the contrary, the rotation and flip data augment methods reduced the classification accuracy. Significance. Decoding motor imagery tasks can benefited from combing the ESI technology and the data augment technology, which is used to solve the problem of low spatial resolution and small samples of EEG signals, respectively. Based on the results, the model proposed has higher accuracy and application potential in the task of MI-EEG binary classification.

Motor Imagery Classification Using Inter-Task Transfer Learning via A Channel-Wise Variational Autoencoder-based Convolutional Neural Network

Highly sophisticated control based on a brain-computer interface (BCI) requires decoding kinematic information from brain signals. The forearm is a region of the upper limb that is often used in everyday life, but intuitive movements within the same limb have rarely been investigated in previous BCI studies. In this study, we focused on various forearm movement decoding from electroencephalography (EEG) signals using a small number of samples. Ten healthy participants took part in an experiment and performed motor execution (ME) and motor imagery (MI) of the intuitive movement tasks (Dataset I). We propose a convolutional neural network using a channel-wise variational autoencoder (CVNet) based on inter-task transfer learning. We approached that training the reconstructed ME-EEG signals together will also achieve more sufficient classification performance with only a small amount of MI-EEG signals. The proposed CVNet was validated on our own Dataset I and a public dataset, BNCI Horizon 2020 (Dataset II). The classification accuracies of various movements are confirmed to be 0.83 (±0.04) and 0.69 (±0.04) for Dataset I and II, respectively. The results show that the proposed method exhibits performance increases of approximately 0.09~0.27 and 0.08~0.24 compared with the conventional models for Dataset I and II, respectively. The outcomes suggest that the training model for decoding imagined movements can be performed using data from ME and a small number of data samples from MI. Hence, it is presented the feasibility of BCI learning strategies that can sufficiently learn deep learning with a few amount of calibration dataset and time only, with stable performance.

From Intricacy to Conciseness: A Progressive Transfer Strategy for EEG-Based Cross-Subject Emotion Recognition

Emotion plays a significant role in human daily activities, and it can be effectively recognized from EEG signals. However, individual variability limits the generalization of emotion classifiers across subjects. Domain adaptation (DA) is a reliable method to solve the issue. Due to the nonstationarity of EEG, the inferior-quality source domain data bring negative transfer in DA procedures. To solve this problem, an auto-augmentation joint distribution adaptation (AA-JDA) method and a burden-lightened and source-preferred JDA (BLSP-JDA) approach are proposed in this paper. The methods are based on a novel transfer idea, learning the specific knowledge of the target domain from the samples that are appropriate for transfer, which reduces the difficulty of transfer between two domains. On multiple emotion databases, our model shows state-of-the-art performance.

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Data Augmentation for Deep Neural Networks Model in EEG Classification Task: A Review

Classification of electroencephalogram (EEG) is a key approach to measure the rhythmic oscillations of neural activity, which is one of the core technologies of brain-computer interface systems (BCIs). However, extraction of the features from non-linear and non-stationary EEG signals is still a challenging task in current algorithms. With the development of artificial intelligence, various advanced algorithms have been proposed for signal classification in recent years. Among them, deep neural networks (DNNs) have become the most attractive type of method due to their end-to-end structure and powerful ability of automatic feature extraction. However, it is difficult to collect large-scale datasets in practical applications of BCIs, which may lead to overfitting or weak generalizability of the classifier. To address these issues, a promising technique has been proposed to improve the performance of the decoding model based on data augmentation (DA). In this article, we investigate recent studies and development of various DA strategies for EEG classification based on DNNs. The review consists of three parts: what kind of paradigms of EEG-based on BCIs are used, what types of DA methods are adopted to improve the DNN models, and what kind of accuracy can be obtained. Our survey summarizes the current practices and performance outcomes that aim to promote or guide the deployment of DA to EEG classification in future research and development.

Realizing the Application of EEG Modeling in BCI Classification: Based on a Conditional GAN Converter

Electroencephalogram (EEG) modeling in brain-computer interface (BCI) provides a theoretical foundation for its development. However, limited by the lack of guidelines in model parameter selection and the inability to obtain personal tissue information in practice, EEG modeling in BCI is mainly focused on the theoretical qualitative level which shows a gap between the theory and its application. Based on such problems, this work combined the surface EEG simulation with a converter based on the generative adversarial network (GAN), to establish the connection from simulated EEG to its application in BCI classification. For the scalp EEGs modeling, a mathematical model was built according to the physics of surface EEG, which consisted of the parallel 3-population neural mass model, the equivalent dipole, and the forward computation. For application, a converter based on the conditional GAN was designed, to transfer the simulated theoretical-only EEG to its practical version, in the lack of individual bio-information. To verify the feasibility, based on the latest microexpression-assisted BCI paradigm proposed by our group, the converted simulated EEGs were used in the training of BCI classifiers. The results indicated that, compared with training with insufficient real data, by adding the simulated EEGs, the overall performance showed a significant improvement (P = 0.04 < 0.05), and the test performance can be improved by 2.17% ± 4.23, in which the largest increase was up to 12.60% ± 1.81. Through this work, the link from theoretical EEG simulation to BCI classification has been initially established, providing an enhanced novel solution for the application of EEG modeling in BCI.

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain-Computer Interfaces

Brain-computer interfaces (BCIs) utilizing machine learning techniques are an emerging technology that enables a communication pathway between a user and an external system, such as a computer. Owing to its practicality, electroencephalography (EEG) is one of the most widely used measurements for BCI. However, EEG has complex patterns and EEG-based BCIs mostly involve a cost/time-consuming calibration phase; thus, acquiring sufficient EEG data is rarely possible. Recently, deep learning (DL) has had a theoretical/practical impact on BCI research because of its use in learning representations of complex patterns inherent in EEG. Moreover, algorithmic advances in DL facilitate short/zero-calibration in BCI, thereby suppressing the data acquisition phase. Those advancements include data augmentation (DA), increasing the number of training samples without acquiring additional data, and transfer learning (TL), taking advantage of representative knowledge obtained from one dataset to address the so-called data insufficiency problem in other datasets. In this study, we review DL-based short/zero-calibration methods for BCI. Further, we elaborate methodological/algorithmic trends, highlight intriguing approaches in the literature, and discuss directions for further research. In particular, we search for generative model-based and geometric manipulation-based DA methods. Additionally, we categorize TL techniques in DL-based BCIs into explicit and implicit methods. Our systematization reveals advances in the DA and TL methods. Among the studies reviewed herein, ~45% of DA studies used generative model-based techniques, whereas ~45% of TL studies used explicit knowledge transferring strategy. Moreover, based on our literature review, we recommend an appropriate DA strategy for DL-based BCIs and discuss trends of TLs used in DL-based BCIs.

Variational Autoencoder for Image-Based Augmentation of Eye-Tracking Data

Over the past decade, deep learning has achieved unprecedented successes in a diversity of application domains, given large-scale datasets. However, particular domains, such as healthcare, inherently suffer from data paucity and imbalance. Moreover, datasets could be largely inaccessible due to privacy concerns, or lack of data-sharing incentives. Such challenges have attached significance to the application of generative modeling and data augmentation in that domain. In this context, this study explores a machine learning-based approach for generating synthetic eye-tracking data. We explore a novel application of variational autoencoders (VAEs) in this regard. More specifically, a VAE model is trained to generate an image-based representation of the eye-tracking output, so-called scanpaths. Overall, our results validate that the VAE model could generate a plausible output from a limited dataset. Finally, it is empirically demonstrated that such approach could be employed as a mechanism for data augmentation to improve the performance in classification tasks.

EEG data augmentation for emotion recognition with a multiple generator conditional Wasserstein GAN

EEG-based emotion recognition has attracted substantial attention from researchers due to its extensive application prospects, and substantial progress has been made in feature extraction and classification modelling from EEG data. However, insufficient high-quality training data are available for building EEG-based emotion recognition models via machine learning or deep learning methods. The artificial generation of high-quality data is an effective approach for overcoming this problem. In this paper, a multi-generator conditional Wasserstein GAN method is proposed for the generation of high-quality artificial that covers a more comprehensive distribution of real data through the use of various generators. Experimental results demonstrate that the artificial data that are generated by the proposed model can effectively improve the performance of emotion classification models that are based on EEG.