Artificial Neural Networks to Assess Emotional States from Brain-Computer Interface

A scoping review on the use of consumer-grade EEG devices for research

BACKGROUND

Commercial electroencephalography (EEG) devices have become increasingly available over the last decade. These devices have been used in a wide variety of fields ranging from engineering to cognitive neuroscience.

PURPOSE

The aim of this study was to chart peer-review articles that used consumer-grade EEG devices to collect neural data. We provide an overview of the research conducted with these relatively more affordable and user-friendly devices. We also inform future research by exploring the current and potential scope of consumer-grade EEG.

METHODS

We followed a five-stage methodological framework for a scoping review that included a systematic search using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines. We searched the following online databases: PsycINFO, MEDLINE, Embase, Web of Science, and IEEE Xplore. We charted study data according to application (BCI, experimental research, validation, signal processing, and clinical) and location of use as indexed by the first author's country.

RESULTS

We identified 916 studies that used data recorded with consumer-grade EEG: 531 were reported in journal articles and 385 in conference papers. Emotiv devices were used most, followed by the NeuroSky MindWave, OpenBCI, interaXon Muse, and MyndPlay Mindband. The most common usage was for brain-computer interfaces, followed by experimental research, signal processing, validation, and clinical purposes.

CONCLUSIONS

Consumer-grade EEG is a useful tool for neuroscientific research and will likely continue to be used well into the future. Our study provides a comprehensive review of their application, as well as future directions for researchers who plan to use these devices.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological States

Affective, emotional, and physiological states (AFFECT) detection and recognition by capturing human signals is a fast-growing area, which has been applied across numerous domains. The research aim is to review publications on how techniques that use brain and biometric sensors can be used for AFFECT recognition, consolidate the findings, provide a rationale for the current methods, compare the effectiveness of existing methods, and quantify how likely they are to address the issues/challenges in the field. In efforts to achieve the key goals of Society 5.0, Industry 5.0, and human-centered design better, the recognition of emotional, affective, and physiological states is progressively becoming an important matter and offers tremendous growth of knowledge and progress in these and other related fields. In this research, a review of AFFECT recognition brain and biometric sensors, methods, and applications was performed, based on Plutchik's wheel of emotions. Due to the immense variety of existing sensors and sensing systems, this study aimed to provide an analysis of the available sensors that can be used to define human AFFECT, and to classify them based on the type of sensing area and their efficiency in real implementations. Based on statistical and multiple criteria analysis across 169 nations, our outcomes introduce a connection between a nation's success, its number of Web of Science articles published, and its frequency of citation on AFFECT recognition. The principal conclusions present how this research contributes to the big picture in the field under analysis and explore forthcoming study trends.

Brain Computer Interface Drone

Brain-Computer Interface has emerged from dazzling experiments of cognitive scientists and researchers who dig deep into the conscious of the human brain where neuroscience, signal processing, machine learning, physical sciences are blended together and neuroprosthesis, neuro spellers, bionic eyes, prosthetic arms, prosthetic legs are created which made the disabled to walk, a mute to express and talk, a blind to see the beautiful world, a deaf to hear, etc. My main aim is to analyze the frequency domain signal of the brain signals of 5 subjects at their respective mental states using an EEG and show how to control a DJI Tello drone using Insight EEG then present the results and interpretation of band power graph, FFT graph and time-domain signals graph of mental commands during the live control of the drone.

Physical Exercise Effects on University Students’ Attention: An EEG Analysis Approach

Physically active breaks (AB) are currently being proposed as an interesting tool to improve students’ attention. Reviews and meta-analyses confirm their effect on attention, but also warned about the sparse evidence based on vigilance and university students. Therefore, this pilot study aimed to (a) determine the effects of AB in comparison with passive breaks on university students’ vigilance and (b) to validate an analysis model based on machine learning algorithms in conjunction with a multiparametric model based on electroencephalography (EEG) signal features. Through a counterbalanced within-subject experimental study, six university students (two female; mean age = 25.67, STD = 3.61) had their vigilance performances (i.e., response time in Psycho-Motor Vigilance Task) and EEG measured, before and after a lecture with an AB and another lecture with a passive break. A multiparametric model based on the spectral power, signal entropy and response time has been developed. Furthermore, this model, together with different machine learning algorithms, shows that for the taken signals there are significant differences after the AB lesson, implying an improvement in attention. These differences are most noticeable with the SVM with RBF kernel and ANNs with F1-score of 85% and 88%, respectively. In conclusion, results showed that students performed better on vigilance after the lecture with AB. Although limited, the evidence found could help researchers to be more accurate in their EEG analyses and lecturers and teachers to improve their students’ attentions in a proper way.

Brain-Computer Interface: Advancement and Challenges

Brain-Computer Interface (BCI) is an advanced and multidisciplinary active research domain based on neuroscience, signal processing, biomedical sensors, hardware, etc. Since the last decades, several groundbreaking research has been conducted in this domain. Still, no comprehensive review that covers the BCI domain completely has been conducted yet. Hence, a comprehensive overview of the BCI domain is presented in this study. This study covers several applications of BCI and upholds the significance of this domain. Then, each element of BCI systems, including techniques, datasets, feature extraction methods, evaluation measurement matrices, existing BCI algorithms, and classifiers, are explained concisely. In addition, a brief overview of the technologies or hardware, mostly sensors used in BCI, is appended. Finally, the paper investigates several unsolved challenges of the BCI and explains them with possible solutions.

Framework for the Classification of Emotions in People With Visual Disabilities Through Brain Signals

Nowadays, the recognition of emotions in people with sensory disabilities still represents a challenge due to the difficulty of generalizing and modeling the set of brain signals. In recent years, the technology that has been used to study a person's behavior and emotions based on brain signals is the brain-computer interface (BCI). Although previous works have already proposed the classification of emotions in people with sensory disabilities using machine learning techniques, a model of recognition of emotions in people with visual disabilities has not yet been evaluated. Consequently, in this work, the authors present a twofold framework focused on people with visual disabilities. Firstly, auditory stimuli have been used, and a component of acquisition and extraction of brain signals has been defined. Secondly, analysis techniques for the modeling of emotions have been developed, and machine learning models for the classification of emotions have been defined. Based on the results, the algorithm with the best performance in the validation is random forest (RF), with an accuracy of 85 and 88% in the classification for negative and positive emotions, respectively. According to the results, the framework is able to classify positive and negative emotions, but the experimentation performed also shows that the framework performance depends on the number of features in the dataset and the quality of the Electroencephalogram (EEG) signals is a determining factor.

Convolutional Neural Network for Drowsiness Detection Using EEG Signals

Drowsiness detection (DD) has become a relevant area of active research in biomedical signal processing. Recently, various deep learning (DL) researches based on the EEG signals have been proposed to detect fatigue conditions. The research presented in this paper proposes an EEG classification system for DD based on DL networks. However, the proposed DD system is mainly realized into two procedures; (i) data acquisition and (ii) model analysis. For the data acquisition procedure, two key steps are considered, which are the signal collection using a wearable Emotiv EPOC+ headset to record 14 channels of EEG, and the signal annotation. Furthermore, a data augmentation (DA) step has been added to the proposed system to overcome the problem of over-fitting and to improve accuracy. As regards the model analysis, a comparative study is also introduced in this paper to argue the choice of DL architecture and frameworks used in our DD system. In this sense, The proposed DD protocol makes use of a convolutional neural network (CNN) architecture implemented using the Keras library. The results showed a high accuracy value (90.42%) in drowsy/awake discrimination and revealed the efficiency of the proposed DD system compared to other research works.

Validation of dynamic virtual faces for facial affect recognition

The ability to recognise facial emotions is essential for successful social interaction. The most common stimuli used when evaluating this ability are photographs. Although these stimuli have proved to be valid, they do not offer the level of realism that virtual humans have achieved. The objective of the present paper is the validation of a new set of dynamic virtual faces (DVFs) that mimic the six basic emotions plus the neutral expression. The faces are prepared to be observed with low and high dynamism, and from front and side views. For this purpose, 204 healthy participants, stratified by gender, age and education level, were recruited for assessing their facial affect recognition with the set of DVFs. The accuracy in responses was compared with the already validated Penn Emotion Recognition Test (ER-40). The results showed that DVFs were as valid as standardised natural faces for accurately recreating human-like facial expressions. The overall accuracy in the identification of emotions was higher for the DVFs (88.25%) than for the ER-40 faces (82.60%). The percentage of hits of each DVF emotion was high, especially for neutral expression and happiness emotion. No statistically significant differences were discovered regarding gender. Nor were significant differences found between younger adults and adults over 60 years. Moreover, there is an increase of hits for avatar faces showing a greater dynamism, as well as front views of the DVFs compared to their profile presentations. DVFs are as valid as standardised natural faces for accurately recreating human-like facial expressions of emotions.

Deep Support Vector Machines for the Identification of Stress Condition from Electrodermal Activity

Early detection of stress condition is beneficial to prevent long-term mental illness like depression and anxiety. This paper introduces an accurate identification of stress/calm condition from electrodermal activity (EDA) signals. The acquisition of EDA signals from a commercial wearable as well as their storage and processing are presented. Several time-domain, frequency-domain and morphological features are extracted over the skin conductance response of the EDA signals. Afterwards, a classification is undergone by using several classical support vector machines (SVMs) and deep support vector machines (D-SVMs). In addition, several binary classifiers are also compared with SVMs in the stress/calm identification task. Moreover, a series of video clips evoking calm and stress conditions have been viewed by 147 volunteers in order to validate the classification results. The highest F1-score obtained for SVMs and D-SVMs are 83% and 92%, respectively. These results demonstrate that not only classical SVMs are appropriate for classification of biomarker signals, but D-SVMs are very competitive in comparison to other classification techniques. In addition, the results have enabled drawing useful considerations for the future use of SVMs and D-SVMs in the specific case of stress/calm identification.

Effect of brightness of visual stimuli on EEG signals

The aim of this study was to examine brightness effect, which is the perceptual property of visual stimuli, on brain responses obtained during visual processing of these stimuli. For this purpose, brain responses of the brain to changes in brightness were explored comparatively using different emotional images (pleasant, unpleasant and neutral) with different luminance levels. In the study, electroencephalography recordings from 12 different electrode sites of 31 healthy participants were used. The power spectra obtained from the analysis of the recordings using short time Fourier transform were analyzed, and a statistical analysis was performed on features extracted from these power spectra. Statistical findings were compared with those obtained from behavioral data. The results showed that the brightness of visual stimuli affected the power of brain responses depending on frequency, time and location. According to the statistically verified findings, the increase in the brightness of pleasant and neutral images increased the average power of responses in the parietal and occipital regions whereas the increase in the brightness of unpleasant images decreased the average power of responses in these regions. Moreover, the statistical results obtained for unpleasant images were found to be in accordance with the behavioral data. The results revealed that the brightness of visual stimuli could be represented by changing the activity power of the brain cortex. The findings emphasized that the brightness of visual stimuli should be viewed as an important parameter in studies using emotional image techniques such as image classification, emotion evaluation and neuro-marketing.

A Computer Mouse Using Blowing Sensors Intended for People with Disabilities

The computer is an important medium that allows people to connect to the internet. However, people with disabilities are unable to use a computer mouse and thus cannot enjoy internet benefits. Nowadays, there are various types of assistive technologies for controlling a computer mouse, but they all have some operational inconveniences. In this paper, we propose an innovative blowing-controlled mouse assistive tool to replace the conventional hand-controlled mouse. Its main contribution is that it uses microphones to induce small signals through the principle of airflow vibration, and it then converts the received signal into the corresponding pulse width. The co-design of software programming enables various mouse functions to be implemented by the identification of the blowing pulse width of multiple microphones. The proposed tool is evaluated experimentally, and the experimental results show that the average identification rate of the proposed mouse is over 85%. Additionally, compared with the other mouse assistive tools, the proposed mouse has the benefits of low cost and humanized operation. Therefore, the proposed blowing control method can not only improve the life quality of people with disabilities but also overcome the disadvantages of existing assistive tools.

Flood Hazard Mapping Using the Flood and Flash-Flood Potential Index in the Buzău River Catchment, Romania

The importance of identifying the areas vulnerable for both floods and flash-floods is an important component of risk management. The assessment of vulnerable areas is a major challenge in the scientific world. The aim of this study is to provide a methodology-oriented study of how to identify the areas vulnerable to floods and flash-floods in the Buzău river catchment by computing two indices: the Flash-Flood Potential Index (FFPI) for the mountainous and the Sub-Carpathian areas, and the Flood Potential Index (FPI) for the low-altitude areas, using the frequency ratio (FR), a bivariate statistical model, the Multilayer Perceptron Neural Networks (MLP), and the ensemble model MLP–FR. A database containing historical flood locations (168 flood locations) and the areas with torrentiality (172 locations with torrentiality) was created and used to train and test the models. The resulting models were computed using GIS techniques, thus resulting the flood and flash-flood vulnerability maps. The results show that the MLP–FR hybrid model had the most performance. The use of the two indices represents a preliminary step in creating flood vulnerability maps, which could represent an important tool for local authorities and a support for flood risk management policies.

A Survey on Deep Learning in Image Polarity Detection: Balancing Generalization Performances and Computational Costs

Deep convolutional neural networks (CNNs) provide an effective tool to extract complex information from images. In the area of image polarity detection, CNNs are customarily utilized in combination with transfer learning techniques to tackle a major problem: the unavailability of large sets of labeled data. Thus, polarity predictors in general exploit a pre-trained CNN as the feature extractor that in turn feeds a classification unit. While the latter unit is trained from scratch, the pre-trained CNN is subject to fine-tuning. As a result, the specific CNN architecture employed as the feature extractor strongly affects the overall performance of the model. This paper analyses state-of-the-art literature on image polarity detection and identifies the most reliable CNN architectures. Moreover, the paper provides an experimental protocol that should allow assessing the role played by the baseline architecture in the polarity detection task. Performance is evaluated in terms of both generalization abilities and computational complexity. The latter attribute becomes critical as polarity predictors, in the era of social networks, might need to be updated within hours or even minutes. In this regard, the paper gives practical hints on the advantages and disadvantages of the examined architectures both in terms of generalization and computational cost.