Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening

Validation of Model-Basis Transfer Learning for a Personalized Electroencephalogram-Based Emotion-Classification Model. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023;2023:1-4. [PMID: 38082699 DOI: 10.1109/embc40787.2023.10340188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

The electroencephalogram (EEG)-based affective brain-computer interface (aBCI) has attracted extensive attention in multidisciplinary fields in the past decade. However, the inherent variability of emotional responses recorded in EEG signals increases the vulnerability of pre-trained machine-learning models and impedes the applicability of aBCIs with real-life settings. To overcome the shortcomings associated with the limited personal data in affective modeling, this study proposes a model-basis transfer learning (TL) approach and verifies its feasibility to construct a personalized model using less emotion-annotated data in a longitudinal eight-day dataset comprising data on 10 subjects. By performing daily reliability testing, the proposed TL approach outperformed the subject-dependent counterpart (using limited data only) by ~6% in binary valence classification after recycling a compact set of the eight most transferable models from other subjects. These empirical findings practically contribute to progress in applying TL in realistic aBCI applications.Clinical Relevance- The proposed model-basis TL approach overcomes the shortcoming of inherent variability in EEG signals, supporting realistic aBCI applications.

Long-Term Exercise Assistance: Group and One-on-One Interactions between a Social Robot and Seniors

For older adults, regular exercises can provide both physical and mental benefits, increase their independence, and reduce the risks of diseases associated with aging. However, only a small portion of older adults regularly engage in physical activity. Therefore, it is important to promote exercise among older adults to help maintain overall health. In this paper, we present the first exploratory long-term human–robot interaction (HRI) study conducted at a local long-term care facility to investigate the benefits of one-on-one and group exercise interactions with an autonomous socially assistive robot and older adults. To provide targeted facilitation, our robot utilizes a unique emotion model that can adapt its assistive behaviors to users’ affect and track their progress towards exercise goals through repeated sessions using the Goal Attainment Scale (GAS), while also monitoring heart rate to prevent overexertion. Results of the study show that users had positive valence and high engagement towards the robot and were able to maintain their exercise performance throughout the study. Questionnaire results showed high robot acceptance for both types of interactions. However, users in the one-on-one sessions perceived the robot as more sociable and intelligent, and had more positive perception of the robot’s appearance and movements.

Enhancing Emotion Recognition Using Region-Specific Electroencephalogram Data and Dynamic Functional Connectivity

Recognizing the emotional states of humans through EEG signals are of great significance to the progress of human-computer interaction. The present study aimed to perform automatic recognition of music-evoked emotions through region-specific information and dynamic functional connectivity of EEG signals and a deep learning neural network. EEG signals of 15 healthy volunteers were collected when different emotions (high-valence-arousal vs. low-valence-arousal) were induced by a musical experimental paradigm. Then a sequential backward selection algorithm combining with deep neural network called Xception was proposed to evaluate the effect of different channel combinations on emotion recognition. In addition, we also assessed whether dynamic functional network of frontal cortex, constructed through different trial number, may affect the performance of emotion cognition. Results showed that the binary classification accuracy based on all 30 channels was 70.19%, the accuracy based on all channels located in the frontal region was 71.05%, and the accuracy based on the best channel combination in the frontal region was 76.84%. In addition, we found that the classification performance increased as longer temporal functional network of frontal cortex was constructed as input features. In sum, emotions induced by different musical stimuli can be recognized by our proposed approach though region-specific EEG signals and time-varying functional network of frontal cortex. Our findings could provide a new perspective for the development of EEG-based emotional recognition systems and advance our understanding of the neural mechanism underlying emotion processing.

A review of research on neuromarketing using content analysis: key approaches and new avenues

There is currently a growing interest in a deeper understanding of consumer behaviour. In this context, the union of different disciplines such as neuroscience and marketing has given birth to new fields of knowledge, e.g. neuromarketing. This study is mainly aimed at carrying out a systematic revision of the literature on neuromarketing from a holistic point of view, analysing its definition and processes, as well as more specific aspects such as its ethics and applications. Based on the results of our review, following a combined methodology with a base dictionary and text mining, our study presents both the current lines of research and the future lines of work.

Recognition of human emotions using EEG signals: A review

Assessment of the cognitive functions and state of clinical subjects is an important aspect of e-health care delivery, and in the development of novel human-machine interfaces. A subject can display a range of emotions that significantly influence cognition, and emotion classification through the analysis of physiological signals is a key means of detecting emotion. Electroencephalography (EEG) signals have become a common focus of such development compared to other physiological signals because EEG employs simple and subject-acceptable methods for obtaining data that can be used for emotion analysis. We have therefore reviewed published studies that have used EEG signal data to identify possible interconnections between emotion and brain activity. We then describe theoretical conceptualization of basic emotions, and interpret the prevailing techniques that have been adopted for feature extraction, selection, and classification. Finally, we have compared the outcomes of these recent studies and discussed the likely future directions and main challenges for researchers developing EEG-based emotion analysis methods.

Analysis of cerebral blood flow entropy while listening to music with emotional content

Objective. Music is one of the most sublime stimuli that human beings can experience. Despite being just an acoustic wave that exerts little physical influence on a subject, it triggers profound changes in emotions and physiological states. This study explores the possibility of detecting subtle changes in cerebral blood flow velocity in response to emotional reactions produced by different musical stimuli using multiscale entropy analysis.Approach. Cerebral blood flow signals were successfully recorded for 16 subjects while performing five different musical tasks. The entropy of each signal was estimated using multiscale sample entropy.Main results. This method has been shown to be capable of revealing the complexity of the internal dynamics of different physiological systems, which cannot be appreciated with classic approaches based on entropy on a single scale.Significance. Significant differences in entropy were found between two of the tasks, which suggests that intense cognitive activities with emotional content cause a decrease in the entropy of cerebral haemodynamics.

Neuroimaging Techniques in Advertising Research: Main Applications, Development, and Brain Regions and Processes

Despite the advancement in neuroimaging tools, studies about using neuroimaging tools to study the impact of advertising on brain regions and processes are scant and remain unclear in academic literature. In this article, we have followed a literature review methodology and a bibliometric analysis to select empirical and review papers that employed neuroimaging tools in advertising campaigns and to understand the global research trends in the neuromarketing domain. We extracted and analyzed sixty-three articles from the Web of Science database to answer our study questions. We found four common neuroimaging techniques employed in advertising research. We also found that the orbitofrontal cortex (OFC), the ventromedial prefrontal cortex, and the dorsolateral prefrontal cortex play a vital role in decision-making processes. The OFC is linked to positive valence, and the lateral OFC and left dorsal anterior insula related in negative valence. In addition, the thalamus and primary visual area associated with the bottom-up attention system, whereas the top-down attention system connected to the dorsolateral prefrontal cortex, parietal cortex, and primary visual areas. For memory, the hippocampus is responsible for generating and processing memories. We hope that this study provides valuable insights about the main brain regions and processes of interest for advertising.

On the estimate of music appraisal from surface EEG: a dynamic-network approach based on cross-sensor PAC measurements

Objective.The aesthetic evaluation of music is strongly dependent on the listener and reflects manifold brain processes that go well beyond the perception of incident sound. Being a high-level cognitive reaction, it is difficult to predict merely from the acoustic features of the audio signal and this poses serious challenges to contemporary music recommendation systems. We attempted to decode music appraisal from brain activity, recorded via wearable EEG, during music listening.Approach.To comply with the dynamic nature of music stimuli, cross-frequency coupling measurements were employed in a time-evolving manner to capture the evolving interactions between distinct brain-rhythms during music listening. Brain response to music was first represented as a continuous flow of functional couplings referring to both regional and inter-regional brain dynamics and then modelled as an ensemble of time-varying (sub)networks. Dynamic graph centrality measures were derived, next, as the final feature-engineering step and, lastly, a support-vector machine was trained to decode the subjective music appraisal. A carefully designed experimental paradigm provided the labeled brain signals.Main results.Using data from 20 subjects, dynamic programming to tailor the decoder to each subject individually and cross-validation, we demonstrated highly satisfactory performance (MAE= 0.948,R²= 0.63) that can be attributed, mostly, to interactions of left frontal gamma rhythm. In addition, our music-appraisal decoder was also employed in a part of the DEAP dataset with similar success. Finally, even a generic version of the decoder (common for all subjects) was found to perform sufficiently.Significance.A novel brain signal decoding scheme was introduced and validated empirically on suitable experimental data. It requires simple operations and leaves room for real-time implementation. Both the code and the experimental data are publicly available.

Hardware Acceleration of EEG-Based Emotion Classification Systems: A Comprehensive Survey

Recent years have witnessed a growing interest in EEG-based wearable classifiers of emotions, which could enable the real-time monitoring of patients suffering from neurological disorders such as Amyotrophic Lateral Sclerosis (ALS), Autism Spectrum Disorder (ASD), or Alzheimer's. The hope is that such wearable emotion classifiers would facilitate the patients' social integration and lead to improved healthcare outcomes for them and their loved ones. Yet in spite of their direct relevance to neuro-medicine, the hardware platforms for emotion classification have yet to fill up some important gaps in their various approaches to emotion classification in a healthcare context. In this paper, we present the first hardware-focused critical review of EEG-based wearable classifiers of emotions and survey their implementation perspectives, their algorithmic foundations, and their feature extraction methodologies. We further provide a neuroscience-based analysis of current hardware accelerators of emotion classifiers and use it to map out several research opportunities, including multi-modal hardware platforms, accelerators with tightly-coupled cores operating robustly in the near/supra-threshold region, and pre-processing libraries for universal EEG-based datasets.

Effective Connectivity During Rest and Music Listening: An EEG Study on Parkinson's Disease

Music-based interventions seem to enhance motor, sensory and cognitive functions in Parkinson’s disease (PD), but the underlying action mechanisms are still largely unknown. This electroencephalography (EEG) study aimed to investigate the effective connectivity patterns characterizing PD in the resting state and during music listening. EEG recordings were obtained from fourteen non-demented PD patients and 12 healthy controls, at rest and while listening to three music tracks. Theta- and alpha-band power spectral density and multivariate partial directed coherence were computed. Power and connectivity measures were compared between patients and controls in the four conditions and in music vs. rest. Compared to controls, patients showed enhanced theta-band power and slightly enhanced alpha-band power, but markedly reduced theta- and alpha-band interactions among EEG channels, especially concerning the information received by the right central channel. EEG power differences were partially reduced by music listening, which induced power increases in controls but not in patients. Connectivity differences were slightly compensated by music, whose effects largely depended on the track. In PD, music enhanced the frontotemporal inter-hemispheric communication. Our findings suggest that PD is characterized by enhanced activity but reduced information flow within the EEG network, being only partially normalized by music. Nevertheless, music capability to facilitate inter-hemispheric communication might underlie its beneficial effects on PD pathophysiology and should be further investigated.

Mathematical Modeling of Brain Activity under Specific Auditory Stimulation

Understanding the connection between different stimuli and the brain response represents a complex research area. However, the use of mathematical models for this purpose is relatively unexplored. The present study investigates the effects of three different auditory stimuli on cerebral biopotentials by means of mathematical functions. The effects of acoustic stimuli (S1, S2, and S3) on cerebral activity were evaluated by electroencephalographic (EEG) recording on 21 subjects for 20 minutes of stimulation, with a 5-minute period of silence before and after stimulation. For the construction of the mathematical models used for the study of the EEG rhythms, we used the Box-Jenkins methodology. Characteristic mathematical models were obtained for the main frequency bands and were expressed by 2 constant functions, 8 first-degree functions, a second-degree function, a fourth-degree function, 6 recursive functions, and 4 periodic functions. The values obtained for the variance estimator are low, demonstrating that the obtained models are correct. The resulting mathematical models allow us to objectively compare the EEG response to the three stimuli, both between the stimuli itself and between each stimulus and the period before stimulation.

A Literature Review of EEG-Based Affective Computing in Marketing

Affect plays an important role in the consumer decision-making process and there is growing interest in the development of new technologies and computational approaches that can interpret and recognize the affects of consumers, with benefits for marketing described in relation to both academia and industry. From an interdisciplinary perspective, this paper aims to review past studies focused on electroencephalography (EEG)-based affective computing (AC) in marketing, which provides a promising avenue for studying the mechanisms underlying affective states and developing recognition computational models to predict the psychological responses of customers. This review offers an introduction to EEG technology and an overview of EEG-based AC; provides a snapshot of the current state of the literature. It briefly presents the themes, challenges, and trends in studies of affect evaluation, recognition, and classification; and further proposes potential guidelines for researchers and marketers.

Is EEG Suitable for Marketing Research? A Systematic Review

Background: In the past decade, marketing studies have greatly benefited from the adoption of neuroscience techniques to explore conscious and unconscious drivers of consumer behavior. Electroencephalography (EEG) is one of the most frequently applied neuroscientific techniques for marketing studies, thanks to its low cost and high temporal resolution. Objective: We present an overview of EEG applications in consumer neuroscience. The aim of this review is to facilitate future research and to highlight reliable approaches for deriving research and managerial implications. Method: We conducted a systematic review by querying five databases for the titles of articles published up to June 2020 with the terms [EEG] AND [neuromarketing] OR [consumer neuroscience]. Results: We screened 264 abstracts and analyzed 113 articles, classified based on research topics (e.g., product characteristics, pricing, advertising attention and memorization, rational, and emotional messages) and characteristics of the experimental design (tasks, stimuli, participants, additional techniques). Conclusions: This review highlights the main applications of EEG to consumer neuroscience research and suggests several ways EEG technique can complement traditional experimental paradigms. Further research areas, including consumer profiling and social consumer neuroscience, have not been sufficiently explored yet and would benefit from EEG techniques to address unanswered questions.

User Affect Elicitation with a Socially Emotional Robot

To effectively communicate with people, social robots must be capable of detecting, interpreting, and responding to human affect during human–robot interactions (HRIs). In order to accurately detect user affect during HRIs, affect elicitation techniques need to be developed to create and train appropriate affect detection models. In this paper, we present such a novel affect elicitation and detection method for social robots in HRIs. Non-verbal emotional behaviors of the social robot were designed to elicit user affect, which was directly measured through electroencephalography (EEG) signals. HRI experiments with both younger and older adults were conducted to evaluate our affect elicitation technique and compare the two types of affect detection models we developed and trained utilizing multilayer perceptron neural networks (NNs) and support vector machines (SVMs). The results showed that; on average, the self-reported valence and arousal were consistent with the intended elicited affect. Furthermore, it was also noted that the EEG data obtained could be used to train affect detection models with the NN models achieving higher classification rates

EEG-Based Emotion Recognition Using Logistic Regression with Gaussian Kernel and Laplacian Prior and Investigation of Critical Frequency Bands

Emotion plays a nuclear part in human attention, decision-making, and communication. Electroencephalogram (EEG)-based emotion recognition has developed a lot due to the application of Brain-Computer Interface (BCI) and its effectiveness compared to body expressions and other physiological signals. Despite significant progress in affective computing, emotion recognition is still an unexplored problem. This paper introduced Logistic Regression (LR) with Gaussian kernel and Laplacian prior for EEG-based emotion recognition. The Gaussian kernel enhances the EEG data separability in the transformed space. The Laplacian prior promotes the sparsity of learned LR regressors to avoid over-specification. The LR regressors are optimized using the logistic regression via variable splitting and augmented Lagrangian (LORSAL) algorithm. For simplicity, the introduced method is noted as LORSAL. Experiments were conducted on the dataset for emotion analysis using EEG, physiological and video signals (DEAP). Various spectral features and features by combining electrodes (power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), rational asymmetry (RASM), and differential caudality (DCAU)) were extracted from different frequency bands (Delta, Theta, Alpha, Beta, Gamma, and Total) with EEG signals. The Naive Bayes (NB), support vector machine (SVM), linear LR with L1-regularization (LR_L1), linear LR with L2-regularization (LR_L2) were used for comparison in the binary emotion classification for valence and arousal. LORSAL obtained the best classification accuracies (77.17% and 77.03% for valence and arousal, respectively) on the DE features extracted from total frequency bands. This paper also investigates the critical frequency bands in emotion recognition. The experimental results showed the superiority of Gamma and Beta bands in classifying emotions. It was presented that DE was the most informative and DASM and DCAU had lower computational complexity with relatively ideal accuracies. An analysis of LORSAL and the recently deep learning (DL) methods is included in the discussion. Conclusions and future work are presented in the final section.

Neural Decoding of Multi-Modal Imagery Behavior Focusing on Temporal Complexity

Mental imagery behaviors of various modalities include visual, auditory, and motor behaviors. Their alterations are pathologically involved in various psychiatric disorders. Results of earlier studies suggest that imagery behaviors are correlated with the modulated activities of the respective modality-specific regions and the additional activities of supramodal imagery-related regions. Additionally, despite the availability of complexity analysis in the neuroimaging field, it has not been used for neural decoding approaches. Therefore, we sought to characterize neural oscillation related to multimodal imagery through complexity-based neural decoding. For this study, we modified existing complexity measures to characterize the time evolution of temporal complexity. We took magnetoencephalography (MEG) data of eight healthy subjects as they performed multimodal imagery and non-imagery tasks. The MEG data were decomposed into amplitude and phase of sub-band frequencies by Hilbert-Huang transform. Subsequently, we calculated the complexity values of each reconstructed time series, along with raw data and band power for comparison, and applied these results as inputs to decode visual perception (VP), visual imagery (VI), motor execution (ME), and motor imagery (MI) functions. Consequently, intra-subject decoding with the complexity yielded a characteristic sensitivity map for each task with high decoding accuracy. The map is inverted in the occipital regions between VP and VI and in the central regions between ME and MI. Additionally, replacement of the labels into two classes as imagery and non-imagery also yielded better classification performance and characteristic sensitivity with the complexity. It is particularly interesting that some subjects showed characteristic sensitivities not only in modality-specific regions, but also in supramodal regions. These analyses indicate that two-class and four-class classifications each provided better performance when using complexity than when using raw data or band power as input. When inter-subject decoding was used with the same model, characteristic sensitivity maps were also obtained, although their decoding performance was lower. Results of this study underscore the availability of complexity measures in neural decoding approaches and suggest the possibility of a modality-independent imagery-related mechanism. The use of time evolution of temporal complexity in neural decoding might extend our knowledge of the neural bases of hierarchical functions in the human brain.

Scaling behaviour in music and cortical dynamics interplay to mediate music listening pleasure

The pleasure of music listening regulates daily behaviour and promotes rehabilitation in healthcare. Human behaviour emerges from the modulation of spontaneous timely coordinated neuronal networks. Too little is known about the physical properties and neurophysiological underpinnings of music to understand its perception, its health benefit and to deploy personalized or standardized music-therapy. Prior studies revealed how macroscopic neuronal and music patterns scale with frequency according to a 1/f^α relationship, where a is the scaling exponent. Here, we examine how this hallmark in music and neuronal dynamics relate to pleasure. Using electroencephalography, electrocardiography and behavioural data in healthy subjects, we show that music listening decreases the scaling exponent of neuronal activity and-in temporal areas-this change is linked to pleasure. Default-state scaling exponents of the most pleased individuals were higher and approached those found in music loudness fluctuations. Furthermore, the scaling in selective regions and timescales and the average heart rate were largely proportional to the scaling of the melody. The scaling behaviour of heartbeat and neuronal fluctuations were associated during music listening. Our results point to a 1/f resonance between brain and music and a temporal rescaling of neuronal activity in the temporal cortex as mechanisms underlying music appreciation.

Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions

Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.

EEG Data Augmentation for Emotion Recognition Using a Conditional Wasserstein GAN

Due to the lack of electroencephalography (EEG) data, it is hard to build an emotion recognition model with high accuracy from EEG signals using machine learning approach. Inspired by generative adversarial networks (GANs), we introduce a Conditional Wasserstein GAN (CWGAN) framework for EEG data augmentation to enhance EEG-based emotion recognition. A Wasserstein GAN with gradient penalty is adopted to generate realistic-like EEG data in differential entropy (DE) form. Three indicators are used to judge the qualities of the generated high-dimensional EEG data, and only high quality data are appended to supplement the data manifold, which leads to better classification of different emotions. We evaluate the proposed CWGAN framework on two public EEG datasets for emotion recognition, namely SEED and DEAP. The experimental results demonstrate that using the EEG data generated by CWGAN significantly improves the accuracies of emotion recognition models.

Constructing a Personalized Cross-Day EEG-Based Emotion-Classification Model Using Transfer Learning

State-of-the-art electroencephalogram (EEG)-based emotion-classification works indicate that a personalized model may not be well exploited until sufficient labeled data are available, given a substantial EEG non-stationarity over days. However, it is impractical to impose a labor-intensive, time-consuming multiple-day data collection. This study proposes a robust principal component analysis (RPCA)-embedded transfer learning (TL) to generate a personalized cross-day model with less labeled data, while obviating intra- and inter-individual differences. Upon the add-session-in validation on two datasets MDME (five-day data of 12 subjects) and SDMN (single-day data of 26 subjects), the experimental results showed that TL enabled the classifier of an MDME individual (using his/her 1st-day session only) to improve progressively in valence and arousal classification by adding similar source sessions (SSs) via the within-dataset TL (wdTL) and cross-dataset TL (cdTL) manners. When recruiting three SSs to test on the 5th-day session, the wdTL improvement (valence: 11.19%, arousal: 5.82%) marginally outperformed the subject-dependent (SD) counterpart (valence: 9.75%, arousal: 3.77%) that was obtained using their own 2nd-4th-day sessions only. The cdTL returned a similar trend in valence (8.35%), yet it was less effective in arousal (0.81%). Most importantly, such cross-day enhancements did not occur in either SD or TL scenarios until RPCA processing. This work sheds light on how to construct a personalized model by leveraging ever-growing EEG repositories.

A Systematic Review for Human EEG Brain Signals Based Emotion Classification, Feature Extraction, Brain Condition, Group Comparison

The study of electroencephalography (EEG) signals is not a new topic. However, the analysis of human emotions upon exposure to music considered as important direction. Although distributed in various academic databases, research on this concept is limited. To extend research in this area, the researchers explored and analysed the academic articles published within the mentioned scope. Thus, in this paper a systematic review is carried out to map and draw the research scenery for EEG human emotion into a taxonomy. Systematically searched all articles about the, EEG human emotion based music in three main databases: ScienceDirect, Web of Science and IEEE Xplore from 1999 to 2016. These databases feature academic studies that used EEG to measure brain signals, with a focus on the effects of music on human emotions. The screening and filtering of articles were performed in three iterations. In the first iteration, duplicate articles were excluded. In the second iteration, the articles were filtered according to their titles and abstracts, and articles outside of the scope of our domain were excluded. In the third iteration, the articles were filtered by reading the full text and excluding articles outside of the scope of our domain and which do not meet our criteria. Based on inclusion and exclusion criteria, 100 articles were selected and separated into five classes. The first class includes 39 articles (39%) consists of emotion, wherein various emotions are classified using artificial intelligence (AI). The second class includes 21 articles (21%) is composed of studies that use EEG techniques. This class is named 'brain condition'. The third class includes eight articles (8%) is related to feature extraction, which is a step before emotion classification. That this process makes use of classifiers should be noted. However, these articles are not listed under the first class because these eight articles focus on feature extraction rather than classifier accuracy. The fourth class includes 26 articles (26%) comprises studies that compare between or among two or more groups to identify and discover human emotion-based EEG. The final class includes six articles (6%) represents articles that study music as a stimulus and its impact on brain signals. Then, discussed the five main categories which are action types, age of the participants, and number size of the participants, duration of recording and listening to music and lastly countries or authors' nationality that published these previous studies. it afterward recognizes the main characteristics of this promising area of science in: motivation of using EEG process for measuring human brain signals, open challenges obstructing employment and recommendations to improve the utilization of EEG process.

Interactive Sonification Exploring Emergent Behavior Applying Models for Biological Information and Listening

Sonification is an open-ended design task to construct sound informing a listener of data. Understanding application context is critical for shaping design requirements for data translation into sound. Sonification requires methodology to maintain reproducibility when data sources exhibit non-linear properties of self-organization and emergent behavior. This research formalizes interactive sonification in an extensible model to support reproducibility when data exhibits emergent behavior. In the absence of sonification theory, extensibility demonstrates relevant methods across case studies. The interactive sonification framework foregrounds three factors: reproducible system implementation for generating sonification; interactive mechanisms enhancing a listener's multisensory observations; and reproducible data from models that characterize emergent behavior. Supramodal attention research suggests interactive exploration with auditory feedback can generate context for recognizing irregular patterns and transient dynamics. The sonification framework provides circular causality as a signal pathway for modeling a listener interacting with emergent behavior. The extensible sonification model adopts a data acquisition pathway to formalize functional symmetry across three subsystems: Experimental Data Source, Sound Generation, and Guided Exploration. To differentiate time criticality and dimensionality of emerging dynamics, tuning functions are applied between subsystems to maintain scale and symmetry of concurrent processes and temporal dynamics. Tuning functions accommodate sonification design strategies that yield order parameter values to render emerging patterns discoverable as well as rehearsable, to reproduce desired instances for clinical listeners. Case studies are implemented with two computational models, Chua's circuit and Swarm Chemistry social agent simulation, generating data in real-time that exhibits emergent behavior. Heuristic Listening is introduced as an informal model of a listener's clinical attention to data sonification through multisensory interaction in a context of structured inquiry. Three methods are introduced to assess the proposed sonification framework: Listening Scenario classification, data flow Attunement, and Sonification Design Patterns to classify sound control. Case study implementations are assessed against these methods comparing levels of abstraction between experimental data and sound generation. Outcomes demonstrate the framework performance as a reference model for representing experimental implementations, also for identifying common sonification structures having different experimental implementations, identifying common functions implemented in different subsystems, and comparing impact of affordances across multiple implementations of listening scenarios.

Improving Cross-Day EEG-Based Emotion Classification Using Robust Principal Component Analysis

Constructing a robust emotion-aware analytical framework using non-invasively recorded electroencephalogram (EEG) signals has gained intensive attentions nowadays. However, as deploying a laboratory-oriented proof-of-concept study toward real-world applications, researchers are now facing an ecological challenge that the EEG patterns recorded in real life substantially change across days (i.e., day-to-day variability), arguably making the pre-defined predictive model vulnerable to the given EEG signals of a separate day. The present work addressed how to mitigate the inter-day EEG variability of emotional responses with an attempt to facilitate cross-day emotion classification, which was less concerned in the literature. This study proposed a robust principal component analysis (RPCA)-based signal filtering strategy and validated its neurophysiological validity and machine-learning practicability on a binary emotion classification task (happiness vs. sadness) using a five-day EEG dataset of 12 subjects when participated in a music-listening task. The empirical results showed that the RPCA-decomposed sparse signals (RPCA-S) enabled filtering off the background EEG activity that contributed more to the inter-day variability, and predominately captured the EEG oscillations of emotional responses that behaved relatively consistent along days. Through applying a realistic add-day-in classification validation scheme, the RPCA-S progressively exploited more informative features (from 12.67 ± 5.99 to 20.83 ± 7.18) and improved the cross-day binary emotion-classification accuracy (from 58.31 ± 12.33% to 64.03 ± 8.40%) as trained the EEG signals from one to four recording days and tested against one unseen subsequent day. The original EEG features (prior to RPCA processing) neither achieved the cross-day classification (the accuracy was around chance level) nor replicated the encouraging improvement due to the inter-day EEG variability. This result demonstrated the effectiveness of the proposed method and may shed some light on developing a realistic emotion-classification analytical framework alleviating day-to-day variability.

Improving EEG-Based Emotion Classification Using Conditional Transfer Learning

To overcome the individual differences, an accurate electroencephalogram (EEG)-based emotion-classification system requires a considerable amount of ecological calibration data for each individual, which is labor-intensive and time-consuming. Transfer learning (TL) has drawn increasing attention in the field of EEG signal mining in recent years. The TL leverages existing data collected from other people to build a model for a new individual with little calibration data. However, brute-force transfer to an individual (i.e., blindly leveraged the labeled data from others) may lead to a negative transfer that degrades performance rather than improving it. This study thus proposed a conditional TL (cTL) framework to facilitate a positive transfer (improving subject-specific performance without increasing the labeled data) for each individual. The cTL first assesses an individual’s transferability for positive transfer and then selectively leverages the data from others with comparable feature spaces. The empirical results showed that among 26 individuals, the proposed cTL framework identified 16 and 14 transferable individuals who could benefit from the data from others for emotion valence and arousal classification, respectively. These transferable individuals could then leverage the data from 18 and 12 individuals who had similar EEG signatures to attain maximal TL improvements in valence- and arousal-classification accuracy. The cTL improved the overall classification performance of 26 individuals by ~15% for valence categorization and ~12% for arousal counterpart, as compared to their default performance based solely on the subject-specific data. This study evidently demonstrated the feasibility of the proposed cTL framework for improving an individual’s default emotion-classification performance given a data repository. The cTL framework may shed light on the development of a robust emotion-classification model using fewer labeled subject-specific data toward a real-life affective brain-computer interface (ABCI).

Effects of Background Music on Objective and Subjective Performance Measures in an Auditory BCI

Several studies have explored brain computer interface (BCI) systems based on auditory stimuli, which could help patients with visual impairments. Usability and user satisfaction are important considerations in any BCI. Although background music can influence emotion and performance in other task environments, and many users may wish to listen to music while using a BCI, auditory, and other BCIs are typically studied without background music. Some work has explored the possibility of using polyphonic music in auditory BCI systems. However, this approach requires users with good musical skills, and has not been explored in online experiments. Our hypothesis was that an auditory BCI with background music would be preferred by subjects over a similar BCI without background music, without any difference in BCI performance. We introduce a simple paradigm (which does not require musical skill) using percussion instrument sound stimuli and background music, and evaluated it in both offline and online experiments. The result showed that subjects preferred the auditory BCI with background music. Different performance measures did not reveal any significant performance effect when comparing background music vs. no background. Since the addition of background music does not impair BCI performance but is preferred by users, auditory (and perhaps other) BCIs should consider including it. Our study also indicates that auditory BCIs can be effective even if the auditory channel is simultaneously otherwise engaged.

Familiarity effects in EEG-based emotion recognition

Although emotion detection using electroencephalogram (EEG) data has become a highly active area of research over the last decades, little attention has been paid to stimulus familiarity, a crucial subjectivity issue. Using both our experimental data and a sophisticated database (DEAP dataset), we investigated the effects of familiarity on brain activity based on EEG signals. Focusing on familiarity studies, we allowed subjects to select the same number of familiar and unfamiliar songs; both resulting datasets demonstrated the importance of reporting self-emotion based on the assumption that the emotional state when experiencing music is subjective. We found evidence that music familiarity influences both the power spectra of brainwaves and the brain functional connectivity to a certain level. We conducted an additional experiment using music familiarity in an attempt to recognize emotional states; our empirical results suggested that the use of only songs with low familiarity levels can enhance the performance of EEG-based emotion classification systems that adopt fractal dimension or power spectral density features and support vector machine, multilayer perceptron or C4.5 classifier. This suggests that unfamiliar songs are most appropriate for the construction of an emotion recognition system.

Using robust principal component analysis to alleviate day-to-day variability in EEG based emotion classification

An emerging challenge for emotion classification using electroencephalography (EEG) is how to effectively alleviate day-to-day variability in raw data. This study employed the robust principal component analysis (RPCA) to address the problem with a posed hypothesis that background or emotion-irrelevant EEG perturbations lead to certain variability across days and somehow submerge emotion-related EEG dynamics. The empirical results of this study evidently validated our hypothesis and demonstrated the RPCA's feasibility through the analysis of a five-day dataset of 12 subjects. The RPCA allowed tackling the sparse emotion-relevant EEG dynamics from the accompanied background perturbations across days. Sequentially, leveraging the RPCA-purified EEG trials from more days appeared to improve the emotion-classification performance steadily, which was not found in the case using the raw EEG features. Therefore, incorporating the RPCA with existing emotion-aware machine-learning frameworks on a longitudinal dataset of each individual may shed light on the development of a robust affective brain-computer interface (ABCI) that can alleviate ecological inter-day variability.

Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls

Estimating cognitive or affective state from neurophysiological signals and designing applications that make use of this information requires expertise in many disciplines such as neurophysiology, machine learning, experimental psychology, and human factors. This makes it difficult to perform research that is strong in all its aspects as well as to judge a study or application on its merits. On the occasion of the special topic "Using neurophysiological signals that reflect cognitive or affective state" we here summarize often occurring pitfalls and recommendations on how to avoid them, both for authors (researchers) and readers. They relate to defining the state of interest, the neurophysiological processes that are expected to be involved in the state of interest, confounding factors, inadvertently "cheating" with classification analyses, insight on what underlies successful state estimation, and finally, the added value of neurophysiological measures in the context of an application. We hope that this paper will support the community in producing high quality studies and well-validated, useful applications.