Electroencephalography Amplitude Modulation Analysis for Automated Affective Tagging of Music Video Clips

Music emotion recognition based on temporal convolutional attention network using EEG

Music is one of the primary ways to evoke human emotions. However, the feeling of music is subjective, making it difficult to determine which emotions music triggers in a given individual. In order to correctly identify emotional problems caused by different types of music, we first created an electroencephalogram (EEG) data set stimulated by four different types of music (fear, happiness, calm, and sadness). Secondly, the differential entropy features of EEG were extracted, and then the emotion recognition model CNN-SA-BiLSTM was established to extract the temporal features of EEG, and the recognition performance of the model was improved by using the global perception ability of the self-attention mechanism. The effectiveness of the model was further verified by the ablation experiment. The classification accuracy of this method in the valence and arousal dimensions is 93.45% and 96.36%, respectively. By applying our method to a publicly available EEG dataset DEAP, we evaluated the generalization and reliability of our method. In addition, we further investigate the effects of different EEG bands and multi-band combinations on music emotion recognition, and the results confirm relevant neuroscience studies. Compared with other representative music emotion recognition works, this method has better classification performance, and provides a promising framework for the future research of emotion recognition system based on brain computer interface.

High and low pitch sound stimuli effects on heart-brain coupling

This study aimed to explore the influence of sound stimulation on heart rate and the potential coupling between cardiac and cerebral activities. Thirty-one participants underwent exposure to periods of silence and two distinct continuous, non-repetitive pure tone stimuli: low pitch (110 Hz) and high pitch (880 Hz). Electroencephalography (EEG) data from electrodes F3, F4, F7, F8, Fp1, Fp2, T3, T4, T5, and T6 were recorded, along with R-R interval data for heart rate. Heart-brain connectivity was assessed using wavelet coherence between heart rate variability (HRV) and EEG envelopes (EEGE). Heart rates were significantly lower during high and low-pitch sound periods than in silence (p < 0.002). HRV-EEGE coherence was significantly lower during high-pitch intervals than silence and low-pitch sound intervals (p < 0.048), specifically between the EEG Beta band and the low-frequency HRV range. These results imply a differential involvement of the frontal and temporal brain regions in response to varying auditory stimuli. Our findings highlight the essential nature of discerning the complex interrelations between sound frequencies and their implications for heart-brain connectivity. Such insights could have ramifications for conditions like seizures and sleep disturbances. A deeper exploration is warranted to decipher specific sound stimuli's potential advantages or drawbacks in diverse clinical scenarios.

EEG Amplitude Modulation Analysis across Mental Tasks: Towards Improved Active BCIs

Brain-computer interface (BCI) technology has emerged as an influential communication tool with extensive applications across numerous fields, including entertainment, marketing, mental state monitoring, and particularly medical neurorehabilitation. Despite its immense potential, the reliability of BCI systems is challenged by the intricacies of data collection, environmental factors, and noisy interferences, making the interpretation of high-dimensional electroencephalogram (EEG) data a pressing issue. While the current trends in research have leant towards improving classification using deep learning-based models, our study proposes the use of new features based on EEG amplitude modulation (AM) dynamics. Experiments on an active BCI dataset comprised seven mental tasks to show the importance of the proposed features, as well as their complementarity to conventional power spectral features. Through combining the seven mental tasks, 21 binary classification tests were explored. In 17 of these 21 tests, the addition of the proposed features significantly improved classifier performance relative to using power spectral density (PSD) features only. Specifically, the average kappa score for these classifications increased from 0.57 to 0.62 using the combined feature set. An examination of the top-selected features showed the predominance of the AM-based measures, comprising over 77% of the top-ranked features. We conclude this paper with an in-depth analysis of these top-ranked features and discuss their potential for use in neurophysiology.

Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study

We have all experienced the sense of time slowing down when we are bored or speeding up when we are focused, engaged, or excited about a task. In virtual reality (VR), perception of time can be a key aspect related to flow, immersion, engagement, and ultimately, to overall quality of experience. While several studies have explored changes in time perception using questionnaires, limited studies have attempted to characterize them objectively. In this paper, we propose the use of a multimodal biosensor-embedded VR headset capable of measuring electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and head movement data while the user is immersed in a virtual environment. Eight gamers were recruited to play a commercial action game comprised of puzzle-solving tasks and first-person shooting and combat. After gameplay, ratings were given across multiple dimensions, including (1) the perception of time flowing differently than usual and (2) the gamers losing sense of time. Several features were extracted from the biosignals, ranked based on a two-step feature selection procedure, and then mapped to a predicted time perception rating using a Gaussian process regressor. Top features were found to come from the four signal modalities and the two regressors, one for each time perception scale, were shown to achieve results significantly better than chance. An in-depth analysis of the top features is presented with the hope that the insights can be used to inform the design of more engaging and immersive VR experiences.

Automated Multi-Wavelength Quality Assessment of Photoplethysmography Signals Using Modulation Spectrum Shape Features

Photoplethysmography (PPG) is used to measure blood volume changes in the microvascular bed of tissue. Information about these changes along time can be used for estimation of various physiological parameters, such as heart rate variability, arterial stiffness, and blood pressure, to name a few. As a result, PPG has become a popular biological modality and is widely used in wearable health devices. However, accurate measurement of various physiological parameters requires good-quality PPG signals. Therefore, various signal quality indexes (SQIs) for PPG signals have been proposed. These metrics have usually been based on statistical, frequency, and/or template analyses. The modulation spectrogram representation, however, captures the second-order periodicities of a signal and has been shown to provide useful quality cues for electrocardiograms and speech signals. In this work, we propose a new PPG quality metric based on properties of the modulation spectrum. The proposed metric is tested using data collected from subjects while they performed various activity tasks contaminating the PPG signals. Experiments on this multi-wavelength PPG dataset show the combination of proposed and benchmark measures significantly outperforming several benchmark SQIs with improvements of 21.3% BACC (balanced accuracy) for green, 21.6% BACC for red, and 19.0% BACC for infrared wavelengths, respectively, for PPG quality detection tasks. The proposed metrics also generalize for cross-wavelength PPG quality detection tasks.

Modulation Spectral Signal Representation for Quality Measurement and Enhancement of Wearable Device Data: A Technical Note

Wearable devices are burgeoning, and applications across numerous verticals are emerging, including human performance monitoring, at-home patient monitoring, and health tracking, to name a few. Off-the-shelf wearables have been developed with focus on portability, usability, and low-cost. As such, when deployed in highly ecological settings, wearable data can be corrupted by artifacts and by missing data, thus severely hampering performance. In this technical note, we overview a signal processing representation called the modulation spectrum. The representation quantifies the rate-of-change of different spectral magnitude components and is shown to separate signal from noise, thus allowing for improved quality measurement, quality enhancement, and noise-robust feature extraction, as well as for disease characterization. We provide an overview of numerous applications developed by the authors over the last decade spanning different wearable modalities and list the results obtained from experimental results alongside comparisons with various state-of-the-art benchmark methods. Open-source software is showcased with the hope that new applications can be developed. We conclude with a discussion on possible future research directions, such as context awareness, signal compression, and improved input representations for deep learning algorithms.

Immersive media experience: a survey of existing methods and tools for human influential factors assessment

Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user's sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.

EEG feature fusion for motor imagery: A new robust framework towards stroke patients rehabilitation

Stroke is the second foremost cause of death worldwide and is one of the most common causes of disability. Several approaches have been proposed to manage stroke patient rehabilitation such as robotic devices and virtual reality systems, and researchers have found that the brain-computer interfaces (BCI) approaches can provide better results. Therefore, the most challenging tasks with BCI applications involve identifying the best technique(s) that can reveal the neuron stimulus information from the patients' brains and extracting the most effective features from these signals as well. Accordingly, the main novelty of this paper is twofold: propose a new feature fusion method for motor imagery (MI)-based BCI and develop an automatic MI framework to detect the changes pre- and post-rehabilitation. This study investigated the electroencephalography (EEG) dataset from post-stroke patients with upper extremity hemiparesis. All patients performed 25 MI-based BCI sessions with follow up assessment visits to examine the functional changes before and after EEG neurorehabilitation. In the first stage, conventional filters and automatic independent component analysis with wavelet transform (AICA-WT) denoising technique were used. Next, attributes from time, entropy and frequency domains were computed, and the effective features were combined into time-entropy-frequency (TEF) attributes. Consequently, the AICA-WT and the TEF fusion set were utilised to develop an AICA-WT-TEF framework. Then, support vector machine (SVM), k-nearest neighbours (kNN) and random forest (RF) classification technique were tested for MI-based BCI rehabilitation. The proposed AICA-WT-TEF framework with RF classifier achieves the best results compared with other classifiers. Finally, the proposed framework and feature fusion set achieve a significant performance in terms of accuracy measures compared to the state-of-the-art. Therefore, the proposed methods could be crucial for improving the process of automatic MI rehabilitation and are recommended for implementation in real-time applications.

Adaptive Filtering for Improved EEG-Based Mental Workload Assessment of Ambulant Users

Recently, due to the emergence of mobile electroencephalography (EEG) devices, assessment of mental workload in highly ecological settings has gained popularity. In such settings, however, motion and other common artifacts have been shown to severely hamper signal quality and to degrade mental workload assessment performance. Here, we show that classical EEG enhancement algorithms, conventionally developed to remove ocular and muscle artifacts, are not optimal in settings where participant movement (e.g., walking or running) is expected. As such, an adaptive filter is proposed that relies on an accelerometer-based referential signal. We show that when combined with classical algorithms, accurate mental workload assessment is achieved. To test the proposed algorithm, data from 48 participants was collected as they performed the Revised Multi-Attribute Task Battery-II (MATB-II) under a low and a high workload setting, either while walking/jogging on a treadmill, or using a stationary exercise bicycle. Accuracy as high as 95% could be achieved with a random forest based mental workload classifier with ambulant users. Moreover, an increase in gamma activity was found in the parietal cortex, suggesting a connection between sensorimotor integration, attention, and workload in ambulant users.

PASS: A Multimodal Database of Physical Activity and Stress for Mobile Passive Body/ Brain-Computer Interface Research

With the burgeoning of wearable devices and passive body/brain-computer interfaces (B/BCIs), automated stress monitoring in everyday settings has gained significant attention recently, with applications ranging from serious games to clinical monitoring. With mobile users, however, challenges arise due to other overlapping (and potentially confounding) physiological responses (e.g., due to physical activity) that may mask the effects of stress, as well as movement artifacts that can be introduced in the measured signals. For example, the classical increase in heart rate can no longer be attributed solely to stress and could be caused by the activity itself. This makes the development of mobile passive B/BCIs challenging. In this paper, we introduce PASS, a multimodal database of Physical Activity and StresS collected from 48 participants. Participants performed tasks of varying stress levels at three different activity levels and provided quantitative ratings of their perceived stress and fatigue levels. To manipulate stress, two video games (i.e., a calm exploration game and a survival game) were used. Peripheral physical activity (electrocardiography, electrodermal activity, breathing, skin temperature) as well as cerebral activity (electroencephalography) were measured throughout the experiment. A complete description of the experimental protocol is provided and preliminary analyses are performed to investigate the physiological reactions to stress in the presence of physical activity. The PASS database, including raw data and subjective ratings has been made available to the research community at http://musaelab.ca/pass-database/. It is hoped that this database will help advance mobile passive B/BCIs for use in everyday settings.

Affective brain-computer interfaces: Choosing a meaningful performance measuring metric

Affective brain-computer interfaces are a relatively new area of research in affective computing. Estimation of affective states can improve human-computer interaction as well as improve the care of people with severe disabilities. To assess the effectiveness of EEG recordings for recognizing affective states, we used data collected in our lab as well as the publicly available DEAP database. We also reviewed the articles that used the DEAP database and found that a significant number of articles did not consider the presence of the class imbalance in the DEAP. Failing to consider class imbalance creates misleading results. Further, ignoring class imbalance makes the comparison of the results between studies using different datasets impossible, since different datasets will have different class imbalances. Class imbalance also shifts the chance level, hence it is vital to consider class bias while determining if the results are above chance. To properly account for the effect of class imbalance, we suggest the use of balanced accuracy as a performance metric, and its posterior distribution for computing credible intervals. For classification, we used features from the literature as well as theta beta-1 ratio. Results from DEAP and our data suggest that the beta band power, theta band power, and theta beta-1 ratio are better feature sets for classifying valence, arousal, and dominance, respectively.

Design of Wearable EEG Devices Specialized for Passive Brain-Computer Interface Applications

Owing to the increased public interest in passive brain–computer interface (pBCI) applications, many wearable devices for capturing electroencephalogram (EEG) signals in daily life have recently been released on the market. However, there exists no well-established criterion to determine the electrode configuration for such devices. Herein, an overall procedure is proposed to determine the optimal electrode configurations of wearable EEG devices that yield the optimal performance for intended pBCI applications. We utilized two EEG datasets recorded in different experiments designed to modulate emotional or attentional states. Emotion-specialized EEG headsets were designed to maximize the accuracy of classification of different emotional states using the emotion-associated EEG dataset, and attention-specialized EEG headsets were designed to maximize the temporal correlation between the EEG index and the behavioral attention index. General purpose electrode configurations were designed to maximize the overall performance in both applications for different numbers of electrodes (2, 4, 6, and 8). The performance was then compared with that of existing wearable EEG devices. Simulations indicated that the proposed electrode configurations allowed for more accurate estimation of the users’ emotional and attentional states than the conventional electrode configurations, suggesting that wearable EEG devices should be designed according to the well-established EEG datasets associated with the target pBCI applications.

Electroencephalogram Profiles for Emotion Identification over the Brain Regions Using Spectral, Entropy and Temporal Biomarkers

Identifying emotions has become essential for comprehending varied human behavior during our daily lives. The electroencephalogram (EEG) has been adopted for eliciting information in terms of waveform distribution over the scalp. The rationale behind this work is twofold. First, it aims to propose spectral, entropy and temporal biomarkers for emotion identification. Second, it aims to integrate the spectral, entropy and temporal biomarkers as a means of developing spectro-spatial ( S S ) , entropy-spatial ( E S ) and temporo-spatial ( T S ) emotional profiles over the brain regions. The EEGs of 40 healthy volunteer students from the University of Vienna were recorded while they viewed seven brief emotional video clips. Features using spectral analysis, entropy method and temporal feature were computed. Three stages of two-way analysis of variance (ANOVA) were undertaken so as to identify the emotional biomarkers and Pearson's correlations were employed to determine the optimal explanatory profiles for emotional detection. The results evidence that the combination of applied spectral, entropy and temporal sets of features may provide and convey reliable biomarkers for identifying S S , E S and T S profiles relating to different emotional states over the brain areas. EEG biomarkers and profiles enable more comprehensive insights into various human behavior effects as an intervention on the brain.

Fusion of Motif- and Spectrum-Related Features for Improved EEG-Based Emotion Recognition

Emotion recognition is a burgeoning field allowing for more natural human-machine interactions and interfaces. Electroencephalography (EEG) has shown to be a useful modality with which user emotional states can be measured and monitored, particularly primitives such as valence and arousal. In this paper, we propose the use of ordinal pattern analysis, also called motifs, for improved EEG-based emotion recognition. Motifs capture recurring structures in time series and are inherently robust to noise, thus are well suited for the task at hand. Several connectivity, asymmetry, and graph-theoretic features are proposed and extracted from the motifs to be used for affective state recognition. Experiments with a widely used public database are conducted, and results show the proposed features outperforming benchmark spectrum-based features, as well as other more recent nonmotif-based graph-theoretic features and amplitude modulation-based connectivity/asymmetry measures. Feature and score-level fusion suggest complementarity between the proposed and benchmark spectrum-based measures. When combined, the fused models can provide up to 9% improvement relative to benchmark features alone and up to 16% to nonmotif-based graph-theoretic features.