1
|
Guenther S, Kosmyna N, Maes P. Image classification and reconstruction from low-density EEG. Sci Rep 2024; 14:16436. [PMID: 39013929 PMCID: PMC11252274 DOI: 10.1038/s41598-024-66228-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 06/28/2024] [Indexed: 07/18/2024] Open
Abstract
Recent advances in visual decoding have enabled the classification and reconstruction of perceived images from the brain. However, previous approaches have predominantly relied on stationary, costly equipment like fMRI or high-density EEG, limiting the real-world availability and applicability of such projects. Additionally, several EEG-based paradigms have utilized artifactual, rather than stimulus-related information yielding flawed classification and reconstruction results. Our goal was to reduce the cost of the decoding paradigm, while increasing its flexibility. Therefore, we investigated whether the classification of an image category and the reconstruction of the image itself is possible from the visually evoked brain activity measured by a portable, 8-channel EEG. To compensate for the low electrode count and to avoid flawed predictions, we designed a theory-guided EEG setup and created a new experiment to obtain a dataset from 9 subjects. We compared five contemporary classification models with our setup reaching an average accuracy of 34.4% for 20 image classes on hold-out test recordings. For the reconstruction, the top-performing model was used as an EEG-encoder which was combined with a pretrained latent diffusion model via double-conditioning. After fine-tuning, we reconstructed images from the test set with a 1000 trial 50-class top-1 accuracy of 35.3%. While not reaching the same performance as MRI-based paradigms on unseen stimuli, our approach greatly improved the affordability and mobility of the visual decoding technology.
Collapse
Affiliation(s)
- Sven Guenther
- School of Computation, Information and Technology, Technical University of Munich, Munich, Germany.
| | - Nataliya Kosmyna
- Media Lab, Massachusetts Institute of Technology, Cambridge, USA
| | - Pattie Maes
- Media Lab, Massachusetts Institute of Technology, Cambridge, USA
| |
Collapse
|
2
|
Mwata-Velu T, Zamora E, Vasquez-Gomez JI, Ruiz-Pinales J, Sossa H. Multiclass Classification of Visual Electroencephalogram Based on Channel Selection, Minimum Norm Estimation Algorithm, and Deep Network Architectures. SENSORS (BASEL, SWITZERLAND) 2024; 24:3968. [PMID: 38931751 PMCID: PMC11207572 DOI: 10.3390/s24123968] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/24/2024] [Revised: 06/04/2024] [Accepted: 06/06/2024] [Indexed: 06/28/2024]
Abstract
This work addresses the challenge of classifying multiclass visual EEG signals into 40 classes for brain-computer interface applications using deep learning architectures. The visual multiclass classification approach offers BCI applications a significant advantage since it allows the supervision of more than one BCI interaction, considering that each class label supervises a BCI task. However, because of the nonlinearity and nonstationarity of EEG signals, using multiclass classification based on EEG features remains a significant challenge for BCI systems. In the present work, mutual information-based discriminant channel selection and minimum-norm estimate algorithms were implemented to select discriminant channels and enhance the EEG data. Hence, deep EEGNet and convolutional recurrent neural networks were separately implemented to classify the EEG data for image visualization into 40 labels. Using the k-fold cross-validation approach, average classification accuracies of 94.8% and 89.8% were obtained by implementing the aforementioned network architectures. The satisfactory results obtained with this method offer a new implementation opportunity for multitask embedded BCI applications utilizing a reduced number of both channels (<50%) and network parameters (<110 K).
Collapse
Affiliation(s)
- Tat’y Mwata-Velu
- Robotics and Mechatronics Lab, Centro de Investigación en Computación, Instituto Politécnico Nacional (CIC–IPN), Avenida Juan de Dios Bátiz esquina Miguel Othón de Mendizábal Colonia Nueva Industrial, Vallejo CP, Gustavo A. Madero, Mexico City 07738, Mexico; (T.M.-V.); (H.S.)
- Section Électricité, Institut Supérieur Pédagogique Technique de Kinshasa (I.S.P.T.-KIN), Av. de la Science 5, Gombe, Kinshasa 03287, Democratic Republic of the Congo
- Telematics and Digital Signal Processing Research Groups (CAs), Department of Electronics Engineering, Universidad de Guanajuato, Salamanca 36885, Mexico (J.R.-P.)
| | - Erik Zamora
- Robotics and Mechatronics Lab, Centro de Investigación en Computación, Instituto Politécnico Nacional (CIC–IPN), Avenida Juan de Dios Bátiz esquina Miguel Othón de Mendizábal Colonia Nueva Industrial, Vallejo CP, Gustavo A. Madero, Mexico City 07738, Mexico; (T.M.-V.); (H.S.)
| | - Juan Irving Vasquez-Gomez
- Centro de Innovación y Desarrollo Tecnológico en Cómputo, Instituto Politécnico Nacional, Avenida Juan de Dios Bátiz esquina Miguel Othón de Mendizábal Colonia Nueva Industrial, Gustavo A. Madero, Mexico City 07738, Mexico;
| | - Jose Ruiz-Pinales
- Telematics and Digital Signal Processing Research Groups (CAs), Department of Electronics Engineering, Universidad de Guanajuato, Salamanca 36885, Mexico (J.R.-P.)
| | - Humberto Sossa
- Robotics and Mechatronics Lab, Centro de Investigación en Computación, Instituto Politécnico Nacional (CIC–IPN), Avenida Juan de Dios Bátiz esquina Miguel Othón de Mendizábal Colonia Nueva Industrial, Vallejo CP, Gustavo A. Madero, Mexico City 07738, Mexico; (T.M.-V.); (H.S.)
| |
Collapse
|
3
|
Zhang S, Wu L, Yu S, Shi E, Qiang N, Gao H, Zhao J, Zhao S. An Explainable and Generalizable Recurrent Neural Network Approach for Differentiating Human Brain States on EEG Dataset. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7339-7350. [PMID: 36331650 DOI: 10.1109/tnnls.2022.3214225] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Electroencephalogram (EEG) is one of the most widely used brain computer interface (BCI) approaches. Despite the success of existing EEG approaches in brain state recognition studies, it is still challenging to differentiate brain states via explainable and generalizable deep learning approaches. In other words, how to explore meaningful and distinguishing features and how to overcome the huge variability and overfitting problem still need to be further studied. To alleviate these challenges, in this work, a multiple random fragment search-based multilayer recurrent neural network (MRFS-MRNN) is proposed to improve the differentiating performance and explore meaningful patterns. Specifically, an explainable MRNN module is proposed to capture the temporal dependences preserved in EEG time series. Besides, a MRFS module is designed to cut multiple random fragments from the entire EEG signal time course to improve the effectiveness of brain state differentiating ability. MRFS-MRNN is concatenatedto effectively overcome the huge variabilities and overfitting problems. Experiment results demonstrate that the proposed MRFS-MRNN model not only has excellent differentiating performance, but also has good explanation and generalization ability. The classification accuracies reach as high as 95.18% for binary classification and 89.19% for four-category classification on the individual level. Similarly, 95.53% and 85.84% classification accuracies are obtained for the binary and four-category classification on the group level. What's more, 94.28% and 85.43% classification accuracies of binary and four-category classifications are achieved for predicting brand new subjects. The experiment results showed that the proposed method outperformed other state-of-the-art (SOTA) models on the same underlying data and improved the explanation and generalization ability.
Collapse
|
4
|
Liang Y, Zhang C, An S, Wang Z, Shi K, Peng T, Ma Y, Xie X, He J, Zheng K. FetchEEG: a hybrid approach combining feature extraction and temporal-channel joint attention for EEG-based emotion classification. J Neural Eng 2024; 21:036011. [PMID: 38701773 DOI: 10.1088/1741-2552/ad4743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2024] [Accepted: 05/03/2024] [Indexed: 05/05/2024]
Abstract
Objective. Electroencephalogram (EEG) analysis has always been an important tool in neural engineering, and the recognition and classification of human emotions are one of the important tasks in neural engineering. EEG data, obtained from electrodes placed on the scalp, represent a valuable resource of information for brain activity analysis and emotion recognition. Feature extraction methods have shown promising results, but recent trends have shifted toward end-to-end methods based on deep learning. However, these approaches often overlook channel representations, and their complex structures pose certain challenges to model fitting.Approach. To address these challenges, this paper proposes a hybrid approach named FetchEEG that combines feature extraction and temporal-channel joint attention. Leveraging the advantages of both traditional feature extraction and deep learning, the FetchEEG adopts a multi-head self-attention mechanism to extract representations between different time moments and channels simultaneously. The joint representations are then concatenated and classified using fully-connected layers for emotion recognition. The performance of the FetchEEG is verified by comparison experiments on a self-developed dataset and two public datasets.Main results. In both subject-dependent and subject-independent experiments, the FetchEEG demonstrates better performance and stronger generalization ability than the state-of-the-art methods on all datasets. Moreover, the performance of the FetchEEG is analyzed for different sliding window sizes and overlap rates in the feature extraction module. The sensitivity of emotion recognition is investigated for three- and five-frequency-band scenarios.Significance. FetchEEG is a novel hybrid method based on EEG for emotion classification, which combines EEG feature extraction with Transformer neural networks. It has achieved state-of-the-art performance on both self-developed datasets and multiple public datasets, with significantly higher training efficiency compared to end-to-end methods, demonstrating its effectiveness and feasibility.
Collapse
Affiliation(s)
- Yu Liang
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Chenlong Zhang
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Shan An
- JD Health International Inc., Beijing, People's Republic of China
| | - Zaitian Wang
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Kaize Shi
- University of Technology Sydney, Sydney, Australia
| | - Tianhao Peng
- Beihang University, Beijing, People's Republic of China
| | - Yuqing Ma
- Beihang University, Beijing, People's Republic of China
| | - Xiaoyang Xie
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Jian He
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Kun Zheng
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| |
Collapse
|
5
|
Kotikam G, Selvaraj L. Golden eagle based improved Att-BiLSTM model for big data classification with hybrid feature extraction and feature selection techniques. NETWORK (BRISTOL, ENGLAND) 2024; 35:154-189. [PMID: 38155542 DOI: 10.1080/0954898x.2023.2293895] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Accepted: 12/06/2023] [Indexed: 12/30/2023]
Abstract
The remarkable development in technology has led to the increase of massive big data. Machine learning processes provide a way for investigators to examine and particularly classify big data. Besides, several machine learning models rely on powerful feature extraction and feature selection techniques for their success. In this paper, a big data classification approach is developed using an optimized deep learning classifier integrated with hybrid feature extraction and feature selection approaches. The proposed technique uses local linear embedding-based kernel principal component analysis and perturbation theory, respectively, to extract more representative data and select the appropriate features from the big data environment. In addition, the feature selection task is fine-tuned by using perturbation theory through heuristic search based on their output accuracy. This feature selection heuristic search method is analysed with five recent heuristic optimization algorithms for deciding the final feature subset. Finally, the data are categorized through an attention-based bidirectional long short-term memory classifier that is optimized with a golden eagle-inspired algorithm. The performance of the proposed model is experimentally verified on publicly accessible datasets. From the experimental outcomes, it is demonstrated that the proposed framework is capable of classifying large datasets with more than 90% accuracy.
Collapse
Affiliation(s)
- Gnanendra Kotikam
- Research Scholar, Department of Information and Communication Engineering, Anna University, Chennai, India
| | - Lokesh Selvaraj
- Department of Computer Science and Engineering, PSG Institute of Technology and Applied Research, Coimbatore, India
| |
Collapse
|
6
|
Huang L, Lv W, Huang Q, Zhang H, Jin S, Chen T, Shen B. Medical equipment effectiveness evaluation model based on cone-constrained DEA and attention-based bi-LSTM. Sci Rep 2024; 14:9324. [PMID: 38654056 DOI: 10.1038/s41598-024-59852-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 04/16/2024] [Indexed: 04/25/2024] Open
Abstract
This study constructs a composite indicator system covering the core dimensions of medical equipment input and output. Based on this system, an innovative cone-constrained data envelopment analysis (DEA) model is designed. The model integrates the advantages of the analytic hierarchy process (AHP) with an improved criterion importance through intercriteria correlation (CRITIC) method to determine subjective and objective weights and employs game theory to obtain the final combined weights, which are further incorporated as constraints to form the cone-constrained DEA model. Finally, a bidirectional long short-term memory (Bi-LSTM) model with an attention mechanism is introduced for integration, aiming to provide a novel and practical model for evaluating the effectiveness of medical equipment. The proposed model has essential reference value for optimizing medical equipment management decision-making and investment strategies.
Collapse
Affiliation(s)
- Luying Huang
- Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200080, China
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China
| | - Wenqian Lv
- Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200080, China
| | - Qingming Huang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China
- School of Medical Imaging, Shanghai University of Medicine and Health Sciences, Shanghai, 201318, China
| | - Haikang Zhang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China
| | - Siyuan Jin
- Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200080, China
| | - Tong Chen
- Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200080, China
| | - Bing Shen
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China.
- Shanghai Tenth People's Hospital of Tongji University, Shanghai, 200072, China.
| |
Collapse
|
7
|
Prasad R, Tarai S, Bit A. Hybrid computational model depicts the contribution of non-significant lobes of human brain during the perception of emotional stimuli. Comput Methods Biomech Biomed Engin 2024:1-27. [PMID: 38328832 DOI: 10.1080/10255842.2024.2311876] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2022] [Accepted: 11/03/2023] [Indexed: 02/09/2024]
Abstract
Emotions are synchronizing responses of human brain while executing cognitive tasks. Earlier studies had revealed strong correlation between specific lobes of the brain to different types of emotional valence. In the current study, a comprehensive three-dimensional mapping of human brain for executing emotion specific tasks had been formulated. A hybrid computational machine learning model customized from Custom Weight Allocation Model (CWAM) and defined as Custom Rank Allocation Model (CRAM). This regression-based hybrid computational model computes the allocated tasks to different lobes of the brain during their respective executive stage. Event Related Potentials (ERP) were obtained with significant effect at P1, P2, P3, N170, N2, and N4. These ERPs were configured at Pz, Cz, F3, and T8 regions of the brain with maximal responses; while regions like Cz, C4 and F4 were also found to make effective contributions to elevate the responses of the brain, and thus these regions were configured as augmented source regions of the brain. In another circumstance of frequent -deviant - equal (FDE) presentation of the emotional stimuli, it was observed that the brain channels C3, C4, P3, P4, O1, O2, and Oz were contributing their emotional quotient to the overall response of the brain regions; whereas, the interaction effect was found presentable at O2, Oz, P3, P4, T8 and C3 regions of brain. The proposed computational model had identified the potential neural pathways during the execution of emotional task.
Collapse
Affiliation(s)
| | | | - Arindam Bit
- Department of Biomedical Engineering, NIT Raipur
| |
Collapse
|
8
|
Xia L, Feng Y, Guo Z, Ding J, Li Y, Li Y, Ma M, Gan G, Xu Y, Luo J, Shi Z, Guan Y. MuLHiTA: A Novel Multiclass Classification Framework With Multibranch LSTM and Hierarchical Temporal Attention for Early Detection of Mental Stress. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:9657-9670. [PMID: 35385389 DOI: 10.1109/tnnls.2022.3159573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Mental stress is an increasingly common psychological issue leading to diseases such as depression, addiction, and heart attack. In this study, an early detection framework based on electroencephalogram (EEG) data is developed for reducing the risk of these diseases. In existing frameworks, signals are often segmented into smaller sections prior to being input to a deep neural network. However, this approach ignores the fundamental nature of EEG signals as a carrier of valuable information (e.g., the integrity of frequency and phase, and temporal fluctuations of EEG components). As such, this type of segmenting may lead to information loss and a failure to effectively identify mental stress levels. Thus, we propose a novel multiclass classification framework termed multibranch LSTM and hierarchical temporal attention (MuLHiTA) for the early identification of mental stress levels. It specifically focuses on not only intraslice (within each slice) but also interslice (between different slices) samples in parallel. This was achieved by including two complementary branches, each of which integrated a specifically designed attention module into a bidirectional long short-term memory (BLSTM) network, enabling extraction of the most discriminative features from interslice and intraslice EEG signals simultaneously. The outputs of attention modules were then summed to obtain a feature representation that contributes to reduce overfitting and more effective multiclass classification. In addition, electrode positions were optimized using neural activity areas under high-stress conditions, thereby reducing computational costs by minimizing the number of critical electrodes. MuLHiTA was evaluated across one private [Montreal imaging stress task (MIST)] and two publicly available EEG datasets [EEG during mental arithmetic tasks (DMAT) and Simultaneous task EEG workload (STEW)]. These were divided into training and test sets using an 8:2 ratio, and the training data were further divided into training and validation sets using a fivefold cross-validation (CV) method, in which the model with the highest accuracy among the five was selected. The model was trained once more with the full training set, and the test data were then used to evaluate its performance. This approach achieved average classification accuracies of 93.58%, 91.80%, and 99.71% for the MIST, STEW, and DMAT datasets, respectively. Experimental results showed MuLHiTA was superior to state-of-the-art algorithms, including EEGNet, BLSTM, EEGLearn, convolutional neural network (CNN)-long short-term memory (LSTM), and convolutional recurrent attention model (CRAM), for multiclass classification. This demonstrates the viability of MuLHiTA for the early detection of mental stress.
Collapse
|
9
|
Zhang S, Shi E, Wu L, Wang R, Yu S, Liu Z, Xu S, Liu T, Zhao S. Differentiating brain states via multi-clip random fragment strategy-based interactive bidirectional recurrent neural network. Neural Netw 2023; 165:1035-1049. [PMID: 37473638 DOI: 10.1016/j.neunet.2023.06.040] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Revised: 05/25/2023] [Accepted: 06/27/2023] [Indexed: 07/22/2023]
Abstract
EEG is widely adopted to study the brain and brain computer interface (BCI) for its non-invasiveness and low costs. Specifically EEG can be applied to differentiate brain states, which is important for better understanding the working mechanisms of the brain. Recurrent neural network (RNN)-based learning strategy has been widely utilized to differentiate brain states, because its optimization architectures improve the classification performance for differentiating brain states at the group level. However, present classification performance is still far from satisfactory. We have identified two major focal points for improvements: one is about organizing the input EEG signals, and the other is related to the design of the RNN architecture. To optimize the above-mentioned issues and achieve better brain state classification performance, we propose a novel multi-clip random fragment strategy-based interactive bidirectional recurrent neural network (McRFS-IBiRNN) model in this work. This model has two advantages over previous methods. First, the McRFS component is designed to re-organize the input EEG signals to make them more suitable for the RNN architecture. Second, the IBiRNN component is an innovative design to model the RNN layers with interaction connections to enhance the fusion of bidirectional features. By adopting the proposed model, promising brain states classification performances are obtained. For example, 96.97% and 99.34% of individual and group level four-category classification accuracies are successfully obtained on the EEG motor/imagery dataset, respectively. A 99.01% accuracy can be observed for four-category classification tasks with new subjects not seen before, which demonstrates the generalization of our proposed method. Compared with existing methods, our model outperforms them with superior results. Overall, the proposed McRFS-IBiRNN model demonstrates great superiority in differentiating brain states on EEG signals.
Collapse
Affiliation(s)
- Shu Zhang
- Center for Brain and Brain-Inspired Computing Research, School of Computer Science, Northwestern Polytechnical University, Xi'an, China.
| | - Enze Shi
- Center for Brain and Brain-Inspired Computing Research, School of Computer Science, Northwestern Polytechnical University, Xi'an, China.
| | - Lin Wu
- School of Automation, Northwestern Polytechnical University, Xi'an, China.
| | - Ruoyang Wang
- Center for Brain and Brain-Inspired Computing Research, School of Computer Science, Northwestern Polytechnical University, Xi'an, China.
| | - Sigang Yu
- Center for Brain and Brain-Inspired Computing Research, School of Computer Science, Northwestern Polytechnical University, Xi'an, China.
| | - Zhengliang Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA.
| | - Shaochen Xu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA.
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA.
| | - Shijie Zhao
- Research and Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen, China.
| |
Collapse
|
10
|
Huang Z, Ma Y, Su J, Shi H, Jia S, Yuan B, Li W, Geng J, Yang T. CDBA: a novel multi-branch feature fusion model for EEG-based emotion recognition. Front Physiol 2023; 14:1200656. [PMID: 37546532 PMCID: PMC10399240 DOI: 10.3389/fphys.2023.1200656] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 07/10/2023] [Indexed: 08/08/2023] Open
Abstract
EEG-based emotion recognition through artificial intelligence is one of the major areas of biomedical and machine learning, which plays a key role in understanding brain activity and developing decision-making systems. However, the traditional EEG-based emotion recognition is a single feature input mode, which cannot obtain multiple feature information, and cannot meet the requirements of intelligent and high real-time brain computer interface. And because the EEG signal is nonlinear, the traditional methods of time domain or frequency domain are not suitable. In this paper, a CNN-DSC-Bi-LSTM-Attention (CDBA) model based on EEG signals for automatic emotion recognition is presented, which contains three feature-extracted channels. The normalized EEG signals are used as an input, the feature of which is extracted by multi-branching and then concatenated, and each channel feature weight is assigned through the attention mechanism layer. Finally, Softmax was used to classify EEG signals. To evaluate the performance of the proposed CDBA model, experiments were performed on SEED and DREAMER datasets, separately. The validation experimental results show that the proposed CDBA model is effective in classifying EEG emotions. For triple-category (positive, neutral and negative) and four-category (happiness, sadness, fear and neutrality), the classification accuracies were respectively 99.44% and 99.99% on SEED datasets. For five classification (Valence 1-Valence 5) on DREAMER datasets, the accuracy is 84.49%. To further verify and evaluate the model accuracy and credibility, the multi-classification experiments based on ten-fold cross-validation were conducted, the elevation indexes of which are all higher than other models. The results show that the multi-branch feature fusion deep learning model based on attention mechanism has strong fitting and generalization ability and can solve nonlinear modeling problems, so it is an effective emotion recognition method. Therefore, it is helpful to the diagnosis and treatment of nervous system diseases, and it is expected to be applied to emotion-based brain computer interface systems.
Collapse
Affiliation(s)
- Zhentao Huang
- School of Electronic Information, Xijing University, Xi’an, China
| | - Yahong Ma
- School of Electronic Information, Xijing University, Xi’an, China
| | - Jianyun Su
- Department of Neurosurgery, Affiliate Children’s Hospital of Xi’an Jiaotong University, Xi’an, China
| | - Hangyu Shi
- Department of Neurosurgery, Affiliate Children’s Hospital of Xi’an Jiaotong University, Xi’an, China
| | - Shanshan Jia
- Department of Neurology, Affiliate Children’s Hospital of Xi’an Jiaotong University, Xi’an, China
| | - Baoxi Yuan
- School of Electronic Information, Xijing University, Xi’an, China
| | - Weisu Li
- School of Electronic Information, Xijing University, Xi’an, China
| | - Jingzhi Geng
- Graduate Student Institute of Xi’an Medical University, Xi’an, Shanxi Province, China
| | - Tingting Yang
- Graduate Student Institute of Xi’an Medical University, Xi’an, Shanxi Province, China
| |
Collapse
|
11
|
Deng Y, Ding S, Li W, Lai Q, Cao L. EEG-based visual stimuli classification via reusable LSTM. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104588] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/15/2023]
|
12
|
A novel personality detection method based on high-dimensional psycholinguistic features and improved distributed Gray Wolf Optimizer for feature selection. Inf Process Manag 2023. [DOI: 10.1016/j.ipm.2022.103217] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
|
13
|
Sibilano E, Brunetti A, Buongiorno D, Lassi M, Grippo A, Bessi V, Micera S, Mazzoni A, Bevilacqua V. An attention-based deep learning approach for the classification of subjective cognitive decline and mild cognitive impairment using resting-state EEG. J Neural Eng 2023; 20. [PMID: 36745929 DOI: 10.1088/1741-2552/acb96e] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Accepted: 02/06/2023] [Indexed: 02/08/2023]
Abstract
Objective. This study aims to design and implement the first deep learning (DL) model to classify subjects in the prodromic states of Alzheimer's disease (AD) based on resting-state electroencephalographic (EEG) signals.Approach. EEG recordings of 17 healthy controls (HCs), 56 subjective cognitive decline (SCD) and 45 mild cognitive impairment (MCI) subjects were acquired at resting state. After preprocessing, we selected sections corresponding to eyes-closed condition. Five different datasets were created by extracting delta, theta, alpha, beta and delta-to-theta frequency bands using bandpass filters. To classify SCDvsMCI and HCvsSCDvsMCI, we propose a framework based on the transformer architecture, which uses multi-head attention to focus on the most relevant parts of the input signals. We trained and validated the model on each dataset with a leave-one-subject-out cross-validation approach, splitting the signals into 10 s epochs. Subjects were assigned to the same class as the majority of their epochs. Classification performances of the transformer were assessed for both epochs and subjects and compared with other DL models.Main results. Results showed that the delta dataset allowed our model to achieve the best performances for the discrimination of SCD and MCI, reaching an Area Under the ROC Curve (AUC) of 0.807, while the highest results for the HCvsSCDvsMCI classification were obtained on alpha and theta with a micro-AUC higher than 0.74.Significance. We demonstrated that DL approaches can support the adoption of non-invasive and economic techniques as EEG to stratify patients in the clinical population at risk for AD. This result was achieved since the attention mechanism was able to learn temporal dependencies of the signal, focusing on the most discriminative patterns, achieving state-of-the-art results by using a deep model of reduced complexity. Our results were consistent with clinical evidence that changes in brain activity are progressive when considering early stages of AD.
Collapse
Affiliation(s)
- Elena Sibilano
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Antonio Brunetti
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Domenico Buongiorno
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Michael Lassi
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy
| | | | - Valentina Bessi
- Department of Neuroscience, Psychology, Drug Research and Child Health, University of Florence, Azienda Ospedaliera Careggi, Florence, Italy
| | - Silvestro Micera
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy.,Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics, Institute of Bioengineering, School of Engineering, Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland
| | - Alberto Mazzoni
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| |
Collapse
|
14
|
Deng X, Wang Z, Liu K, Xiang X. A GAN model encoded by CapsEEGNet for visual EEG encoding and image reproduction. J Neurosci Methods 2023; 384:109747. [PMID: 36427669 DOI: 10.1016/j.jneumeth.2022.109747] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2022] [Revised: 10/27/2022] [Accepted: 11/11/2022] [Indexed: 11/25/2022]
Abstract
In last few decades, reading the human mind is an innovative topic in scientific research. Recent studies in neuroscience indicate that it is possible to decode the signals of the human brain based on the neuroimaging data. The work in this paper explores the possibility of building an end-to-end BCI system to learn and visualize the brain thoughts evoked by the stimulating images. To achieve this goal, it designs an experiment to collect the EEG signals evoked by randomly presented images. Based on these data, this work analyzes and compares the classification abilities by several improved methods, including the Transformer, CapsNet and the ensemble strategies. After obtaining the optimal method to be the encoder, this paper proposes a distribution-to-distribution mapping network to transform an encoded latent feature vector into a prior image feature vector. To visualize the brain thoughts, a pretrained IC-GAN model is used to receive these image feature vectors and generate images. Extensive experiments are carried out and the results show that the proposed method can effectively deal with the small sample data original from the less electrode channels. By examining the generated images coming from the EEG signals, it verifies that the proposed model is capable of reproducing the images seen by human eyes to some extent.
Collapse
Affiliation(s)
- Xin Deng
- Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 40065, China.
| | - Zhongyin Wang
- Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 40065, China.
| | - Ke Liu
- Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 40065, China.
| | - Xiaohong Xiang
- Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 40065, China.
| |
Collapse
|
15
|
Dogan A, Barua PD, Baygin M, Tuncer T, Dogan S, Yaman O, Dogru AH, Acharya RU. Automated accurate emotion classification using Clefia pattern-based features with EEG signals. INTERNATIONAL JOURNAL OF HEALTHCARE MANAGEMENT 2022. [DOI: 10.1080/20479700.2022.2141694] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
Affiliation(s)
- Abdullah Dogan
- Department of Computer Engineering, Middle East Technical University, Ankara, Turkey
| | - Prabal Datta Barua
- School of Business (Information System), University of Southern Queensland, Toowoomba, Australia
- Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Turkey
| | - Mehmet Baygin
- Department of Computer Engineering, College of Engineering, Ardahan University, Ardahan, Turkey
| | - Turker Tuncer
- Department of Digital Forensics Engineering, College of Technology, Firat University, Elazig, Turkey
| | - Sengul Dogan
- Department of Digital Forensics Engineering, College of Technology, Firat University, Elazig, Turkey
| | - Orhan Yaman
- Department of Digital Forensics Engineering, College of Technology, Firat University, Elazig, Turkey
| | - Ali Hikmet Dogru
- Department of Computer Engineering, Middle East Technical University, Ankara, Turkey
| | - Rajendra U. Acharya
- Ngee Ann Polytechnic, Department of Electronics and Computer Engineering, Singapore
- Department of Biomedical Engineering, School of Science and Technology, SUSS University, Singapore
- Department of Biomedical Informatics and Medical Engineering, Asia University, Taichung, Taiwan
| |
Collapse
|
16
|
EEG Signal Analysis based on machine learning in psychological balance detection of athletes. EVOLUTIONARY INTELLIGENCE 2022. [DOI: 10.1007/s12065-022-00770-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
17
|
Spatiotemporal grid-based crash prediction—application of a transparent deep hybrid modeling framework. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07511-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
18
|
Dai Y, Zhou Q, Leng M, Yang X, Wang Y. Improving the Bi-LSTM model with XGBoost and attention mechanism: A combined approach for short-term power load prediction. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2022.109632] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
19
|
Fei Y, Zhang H, Wang Y, Liu Z, Liu Y. LTPConstraint: a transfer learning based end-to-end method for RNA secondary structure prediction. BMC Bioinformatics 2022; 23:354. [PMID: 35999499 PMCID: PMC9396797 DOI: 10.1186/s12859-022-04847-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 07/18/2022] [Indexed: 11/26/2022] Open
Abstract
Background RNA secondary structure is very important for deciphering cell’s activity and disease occurrence. The first method which was used by the academics to predict this structure is biological experiment, But this method is too expensive, causing the promotion to be affected. Then, computing methods emerged, which has good efficiency and low cost. However, the accuracy of computing methods are not satisfactory. Many machine learning methods have also been applied to this area, but the accuracy has not improved significantly. Deep learning has matured and achieves great success in many areas such as computer vision and natural language processing. It uses neural network which is a kind of structure that has good functionality and versatility, but its effect is highly correlated with the quantity and quality of the data. At present, there is no model with high accuracy, low data dependence and high convenience in predicting RNA secondary structure. Results This paper designs a neural network called LTPConstraint to predict RNA secondary structure. The network is based on many network structure such as Bidirectional LSTM, Transformer and generator. It also uses transfer learning to train modelso that the data dependence can be reduced. Conclusions LTPConstraint has achieved high accuracy in RNA secondary structure prediction. Compared with the previous methods, the accuracy improves obviously both in predicting the structure with pseudoknot and the structure without pseudoknot. At the same time, LTPConstraint is easy to operate and can achieve result very quickly.
Collapse
Affiliation(s)
- Yinchao Fei
- College of Computer Science and Technology, Jilin University, Changchun, China.,Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University, Changchun, China
| | - Hao Zhang
- College of Computer Science and Technology, Jilin University, Changchun, China.,Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University, Changchun, China
| | - Yili Wang
- College of Computer Science and Technology, Jilin University, Changchun, China.,Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University, Changchun, China
| | - Zhen Liu
- Graduate School of Engineering, Nagasaki Institute of Applied Science, Nagasaki, Japan
| | - Yuanning Liu
- College of Computer Science and Technology, Jilin University, Changchun, China. .,Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University, Changchun, China.
| |
Collapse
|
20
|
Identifying and Classifying Enhancers by Dinucleotide-Based Auto-Cross Covariance and Attention-Based Bi-LSTM. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2022; 2022:7518779. [PMID: 35422876 PMCID: PMC9005296 DOI: 10.1155/2022/7518779] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 03/12/2022] [Indexed: 11/17/2022]
Abstract
Enhancers are a class of noncoding DNA elements located near structural genes. In recent years, their identification and classification have been the focus of research in the field of bioinformatics. However, due to their high free scattering and position variability, although the performance of the prediction model has been continuously improved, there is still a lot of room for progress. In this paper, density-based spatial clustering of applications with noise (DBSCAN) was used to screen the physicochemical properties of dinucleotides to extract dinucleotide-based auto-cross covariance (DACC) features; then, the features are reduced by feature selection Python toolkit MRMD 2.0. The reduced features are input into the random forest to identify enhancers. The enhancer classification model was built by word2vec and attention-based Bi-LSTM. Finally, the accuracies of our enhancer identification and classification models were 77.25% and 73.50%, respectively, and the Matthews’ correlation coefficients (MCCs) were 0.5470 and 0.4881, respectively, which were better than the performance of most predictors.
Collapse
|
21
|
Decoding Selective Auditory Attention with EEG using A Transformer Model. Methods 2022; 204:410-417. [DOI: 10.1016/j.ymeth.2022.04.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Revised: 03/20/2022] [Accepted: 04/14/2022] [Indexed: 11/23/2022] Open
|
22
|
Zhang C, Sun L, Ge S, Chang Y, Jin M, Xiao Y, Gao H, Wang L, Cong F. Quantitative evaluation of short-term resting-state brain networks for primary insomnia diagnosis. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103498] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
23
|
Abstract
As a long-standing research topic in the field of brain–computer interface, emotion recognition still suffers from low recognition accuracy. In this research, we present a novel model named DE-CNN-BiLSTM deeply integrating the complexity of EEG signals, the spatial structure of brain and temporal contexts of emotion formation. Firstly, we extract the complexity properties of the EEG signal by calculating Differential Entropy in different time slices of different frequency bands to obtain 4D feature tensors according to brain location. Subsequently, the 4D tensors are input into the Convolutional Neural Network to learn brain structure and output time sequences; after that Bidirectional Long-Short Term Memory is used to learn past and future information of the time sequences. Compared with the existing emotion recognition models, the new model can decode the EEG signal deeply and extract key emotional features to improve accuracy. The simulation results show the algorithm achieves an average accuracy of 94% for DEAP dataset and 94.82% for SEED dataset, confirming its high accuracy and strong robustness.
Collapse
|