1
|
Li M, Qi E, Xu G, Jin J, Zhao Q, Guo M, Liao W. A delayed matching task-based study on action sequence of motor imagery. Cogn Neurodyn 2024; 18:1593-1607. [PMID: 39104677 PMCID: PMC11297855 DOI: 10.1007/s11571-023-10030-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Revised: 09/28/2023] [Accepted: 10/23/2023] [Indexed: 08/07/2024] Open
Abstract
The way people imagine greatly affects performance of brain-computer interface (BCI) based on motion imagery (MI). Action sequence is a basic unit of imitation, learning, and memory for motor behavior. Whether it influences the MI-BCI is unknown, and how to manifest this influence is difficult since the MI is a spontaneous brain activity. To investigate the influence of the action sequence, this study proposes a novel paradigm named action sequences observing and delayed matching task to use images and videos to guide people to observe, match and reinforce the memory of sequence. Seven subjects' ERPs and MI performance are analyzed under four different levels of complexities or orders of the sequence. Results demonstrated that the action sequence in terms of complexity and sequence order significantly affects the MI. The complex action in positive order obtains stronger ERD/ERS and more pronounced MI feature distributions, and yields an MI classification accuracy that is 12.3% higher than complex action in negative order (p < 0.05). In addition, the ERP amplitudes derived from the supplementary motor area show a positive correlation to the MI. This study demonstrates a new perspective of improving imagery in the MI-BCI by considering the complexity and order of the action sequences, and provides a novel index for manifesting the MI performance by ERP.
Collapse
Affiliation(s)
- Mengfan Li
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, 300132 China
- Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Tianjin, 300132 China
- Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin, 300132 China
| | - Enming Qi
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, 300132 China
- Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Tianjin, 300132 China
- Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin, 300132 China
| | - Guizhi Xu
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, 300132 China
- Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Tianjin, 300132 China
- Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin, 300132 China
| | - Jing Jin
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, East China, University of Science and Technology, Shanghai, 518063 China
- Shenzhen Research Institute of East China, University of Science and Technology, Shenzhen, 518063 China
| | - Qi Zhao
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, 300132 China
- Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Tianjin, 300132 China
- Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin, 300132 China
| | - Miaomiao Guo
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, 300132 China
- Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Tianjin, 300132 China
- Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin, 300132 China
| | - Wenzhe Liao
- School of Artificial Intelligence, Hebei University of Technology, Tianjin, 300132 China
| |
Collapse
|
2
|
Zhao X, Xu R, Xu R, Wang X, Cichocki A, Jin J. An auto-segmented multi-time window dual-scale neural network for brain-computer interfaces based on event-related potentials. J Neural Eng 2024; 21:046008. [PMID: 38848710 DOI: 10.1088/1741-2552/ad558a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 06/07/2024] [Indexed: 06/09/2024]
Abstract
Objective.Event-related potentials (ERPs) are cerebral responses to cognitive processes, also referred to as cognitive potentials. Accurately decoding ERPs can help to advance research on brain-computer interfaces (BCIs). The spatial pattern of ERP varies with time. In recent years, convolutional neural networks (CNNs) have shown promising results in electroencephalography (EEG) classification, specifically for ERP-based BCIs.Approach.This study proposes an auto-segmented multi-time window dual-scale neural network (AWDSNet). The combination of a multi-window design and a lightweight base network gives AWDSNet good performance at an acceptable cost of computing. For each individual, we create a time window set by calculating the correlation of signedR-squared values, which enables us to determine the length and number of windows automatically. The signal data are segmented based on the obtained window sets in sub-plus-global mode. Then, the multi-window data are fed into a dual-scale CNN model, where the sizes of the convolution kernels are determined by the window sizes. The use of dual-scale spatiotemporal convolution focuses on feature details while also having a large enough receptive length, and the grouping parallelism undermines the increase in the number of parameters that come with dual scaling.Main results.We evaluated the performance of AWDSNet on a public dataset and a self-collected dataset. A comparison was made with four popular methods including EEGNet, DeepConvNet, EEG-Inception, and PPNN. The experimental results show that AWDSNet has excellent classification performance with acceptable computational complexity.Significance.These results indicate that AWDSNet has great potential for applications in ERP decoding.
Collapse
Affiliation(s)
- Xueqing Zhao
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, People's Republic of China
| | - Ren Xu
- g.tec medical engineering GmbH, Schiedlberg, Austria
| | - Ruitian Xu
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, People's Republic of China
| | - Xingyu Wang
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, People's Republic of China
| | - Andrzej Cichocki
- Systems Research Institute of Polish Academy of Science, 01-447 Warsaw, Poland
- Tokyo University of Agriculture and Technology, Tokyo 184-8588184-8588, Japan
- RIKEN Advanced Intelligence Project, Tokyo 103-0027, Japan
| | - Jing Jin
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, People's Republic of China
| |
Collapse
|
3
|
Zhao S, Fang L, Yang Y, Tang G, Luo G, Han J, Liu T, Hu X. Task sub-type states decoding via group deep bidirectional recurrent neural network. Med Image Anal 2024; 94:103136. [PMID: 38489895 DOI: 10.1016/j.media.2024.103136] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 01/31/2024] [Accepted: 03/05/2024] [Indexed: 03/17/2024]
Abstract
Decoding brain states under different cognitive tasks from functional magnetic resonance imaging (fMRI) data has attracted great attention in the neuroimaging filed. However, the well-known temporal dependency in fMRI sequences has not been fully exploited in existing studies, due to the limited temporal-modeling capacity of the backbone machine learning algorithms and rigid training sample organization strategies upon which the brain decoding methods are built. To address these limitations, we propose a novel method for fine-grain brain state decoding, namely, group deep bidirectional recurrent neural network (Group-DBRNN) model. We first propose a training sample organization strategy that consists of a group-task sample generation module and a multiple-scale random fragment strategy (MRFS) module to collect training samples that contain rich task-relevant brain activity contrast (i.e., the comparison of neural activity patterns between different tasks) and maintain the temporal dependency. We then develop a novel decoding model by replacing the unidirectional RNNs that are widely used in existing brain state decoding studies with bidirectional stacked RNNs to better capture the temporal dependency, and by introducing a multi-task interaction layer (MTIL) module to effectively model the task-relevant brain activity contrast. Our experimental results on the Human Connectome Project task fMRI dataset (7 tasks consisting of 23 task sub-type states) show that the proposed model achieves an average decoding accuracy of 94.7% over the 23 fine-grain sub-type states. Meanwhile, our extensive interpretations of the intermediate features learned in the proposed model via visualizations and quantitative assessments of their discriminability and inter-subject alignment evidence that the proposed model can effectively capture the temporal dependency and task-relevant contrast.
Collapse
Affiliation(s)
- Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China; Research & Development Institute of Northwestern Polytechnical University in Shenzhen, China
| | - Long Fang
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China
| | - Yang Yang
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China
| | - Guochang Tang
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China
| | - Guoxin Luo
- Department of Ophthalmology, Nanyang First People's Hospital Affiliated to Henan University, Nanyang 473000, China
| | - Junwei Han
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China
| | - Tianming Liu
- School of Computing, The University of Georgia, GA, USA
| | - Xintao Hu
- School of Automation, Northwestern Polytechnical University, Xi'an 710072, China.
| |
Collapse
|
4
|
Ferrero L, Soriano-Segura P, Navarro J, Jones O, Ortiz M, Iáñez E, Azorín JM, Contreras-Vidal JL. Brain-machine interface based on deep learning to control asynchronously a lower-limb robotic exoskeleton: a case-of-study. J Neuroeng Rehabil 2024; 21:48. [PMID: 38581031 PMCID: PMC10996198 DOI: 10.1186/s12984-024-01342-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Accepted: 03/15/2024] [Indexed: 04/07/2024] Open
Abstract
BACKGROUND This research focused on the development of a motor imagery (MI) based brain-machine interface (BMI) using deep learning algorithms to control a lower-limb robotic exoskeleton. The study aimed to overcome the limitations of traditional BMI approaches by leveraging the advantages of deep learning, such as automated feature extraction and transfer learning. The experimental protocol to evaluate the BMI was designed as asynchronous, allowing subjects to perform mental tasks at their own will. METHODS A total of five healthy able-bodied subjects were enrolled in this study to participate in a series of experimental sessions. The brain signals from two of these sessions were used to develop a generic deep learning model through transfer learning. Subsequently, this model was fine-tuned during the remaining sessions and subjected to evaluation. Three distinct deep learning approaches were compared: one that did not undergo fine-tuning, another that fine-tuned all layers of the model, and a third one that fine-tuned only the last three layers. The evaluation phase involved the exclusive closed-loop control of the exoskeleton device by the participants' neural activity using the second deep learning approach for the decoding. RESULTS The three deep learning approaches were assessed in comparison to an approach based on spatial features that was trained for each subject and experimental session, demonstrating their superior performance. Interestingly, the deep learning approach without fine-tuning achieved comparable performance to the features-based approach, indicating that a generic model trained on data from different individuals and previous sessions can yield similar efficacy. Among the three deep learning approaches compared, fine-tuning all layer weights demonstrated the highest performance. CONCLUSION This research represents an initial stride toward future calibration-free methods. Despite the efforts to diminish calibration time by leveraging data from other subjects, complete elimination proved unattainable. The study's discoveries hold notable significance for advancing calibration-free approaches, offering the promise of minimizing the need for training trials. Furthermore, the experimental evaluation protocol employed in this study aimed to replicate real-life scenarios, granting participants a higher degree of autonomy in decision-making regarding actions such as walking or stopping gait.
Collapse
Affiliation(s)
- Laura Ferrero
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Elche, Spain.
- Instituto de Investigación en Ingeniería de Elche-I3E, Miguel Hernández University of Elche, Elche, Spain.
- International Affiliate NSF IUCRC BRAIN Site, Miguel Hernández University of Elche, Elche, Spain.
- NSF IUCRC BRAIN, University of Houston, Houston, USA.
- Non-Invasive Brain Machine Interface Systems, University of Houston, Houston, TX, USA.
| | - Paula Soriano-Segura
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Elche, Spain
- Instituto de Investigación en Ingeniería de Elche-I3E, Miguel Hernández University of Elche, Elche, Spain
- International Affiliate NSF IUCRC BRAIN Site, Miguel Hernández University of Elche, Elche, Spain
| | - Jacobo Navarro
- NSF IUCRC BRAIN, University of Houston, Houston, USA
- International Affiliate NSF IUCRC BRAIN Site, Tecnológico de Monterrey, Monterrey, Mexico
- Non-Invasive Brain Machine Interface Systems, University of Houston, Houston, TX, USA
| | - Oscar Jones
- NSF IUCRC BRAIN, University of Houston, Houston, USA
- Non-Invasive Brain Machine Interface Systems, University of Houston, Houston, TX, USA
| | - Mario Ortiz
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Elche, Spain
- Instituto de Investigación en Ingeniería de Elche-I3E, Miguel Hernández University of Elche, Elche, Spain
- International Affiliate NSF IUCRC BRAIN Site, Miguel Hernández University of Elche, Elche, Spain
| | - Eduardo Iáñez
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Elche, Spain
- Instituto de Investigación en Ingeniería de Elche-I3E, Miguel Hernández University of Elche, Elche, Spain
- International Affiliate NSF IUCRC BRAIN Site, Miguel Hernández University of Elche, Elche, Spain
| | - José M Azorín
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Elche, Spain
- Instituto de Investigación en Ingeniería de Elche-I3E, Miguel Hernández University of Elche, Elche, Spain
- International Affiliate NSF IUCRC BRAIN Site, Miguel Hernández University of Elche, Elche, Spain
- Valencian Graduate School and Research Network of Artificial Intelligence-valgrAI, Valencia, Spain
| | - José L Contreras-Vidal
- NSF IUCRC BRAIN, University of Houston, Houston, USA
- Non-Invasive Brain Machine Interface Systems, University of Houston, Houston, TX, USA
| |
Collapse
|
5
|
Xue Q, Song Y, Wu H, Cheng Y, Pan H. Graph neural network based on brain inspired forward-forward mechanism for motor imagery classification in brain-computer interfaces. Front Neurosci 2024; 18:1309594. [PMID: 38606308 PMCID: PMC11008472 DOI: 10.3389/fnins.2024.1309594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Accepted: 03/04/2024] [Indexed: 04/13/2024] Open
Abstract
Introduction Within the development of brain-computer interface (BCI) systems, it is crucial to consider the impact of brain network dynamics and neural signal transmission mechanisms on electroencephalogram-based motor imagery (MI-EEG) tasks. However, conventional deep learning (DL) methods cannot reflect the topological relationship among electrodes, thereby hindering the effective decoding of brain activity. Methods Inspired by the concept of brain neuronal forward-forward (F-F) mechanism, a novel DL framework based on Graph Neural Network combined forward-forward mechanism (F-FGCN) is presented. F-FGCN framework aims to enhance EEG signal decoding performance by applying functional topological relationships and signal propagation mechanism. The fusion process involves converting the multi-channel EEG into a sequence of signals and constructing a network grounded on the Pearson correlation coeffcient, effectively representing the associations between channels. Our model initially pre-trains the Graph Convolutional Network (GCN), and fine-tunes the output layer to obtain the feature vector. Moreover, the F-F model is used for advanced feature extraction and classification. Results and discussion Achievement of F-FGCN is assessed on the PhysioNet dataset for a four-class categorization, compared with various classical and state-of-the-art models. The learned features of the F-FGCN substantially amplify the performance of downstream classifiers, achieving the highest accuracy of 96.11% and 82.37% at the subject and group levels, respectively. Experimental results affirm the potency of FFGCN in enhancing EEG decoding performance, thus paving the way for BCI applications.
Collapse
Affiliation(s)
- Qiwei Xue
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
- University of Science and Technology of China, Hefei, China
- Mechanical Department, School of Energy Systems, Lappeenranta University of Technology (LUT), Lappeenranta, Finland
| | - Yuntao Song
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
- University of Science and Technology of China, Hefei, China
| | - Huapeng Wu
- Mechanical Department, School of Energy Systems, Lappeenranta University of Technology (LUT), Lappeenranta, Finland
| | - Yong Cheng
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
| | - Hongtao Pan
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
| |
Collapse
|
6
|
Rakhmatulin I, Dao MS, Nassibi A, Mandic D. Exploring Convolutional Neural Network Architectures for EEG Feature Extraction. SENSORS (BASEL, SWITZERLAND) 2024; 24:877. [PMID: 38339594 PMCID: PMC10856895 DOI: 10.3390/s24030877] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Revised: 01/12/2024] [Accepted: 01/20/2024] [Indexed: 02/12/2024]
Abstract
The main purpose of this paper is to provide information on how to create a convolutional neural network (CNN) for extracting features from EEG signals. Our task was to understand the primary aspects of creating and fine-tuning CNNs for various application scenarios. We considered the characteristics of EEG signals, coupled with an exploration of various signal processing and data preparation techniques. These techniques include noise reduction, filtering, encoding, decoding, and dimension reduction, among others. In addition, we conduct an in-depth analysis of well-known CNN architectures, categorizing them into four distinct groups: standard implementation, recurrent convolutional, decoder architecture, and combined architecture. This paper further offers a comprehensive evaluation of these architectures, covering accuracy metrics, hyperparameters, and an appendix that contains a table outlining the parameters of commonly used CNN architectures for feature extraction from EEG signals.
Collapse
Affiliation(s)
- Ildar Rakhmatulin
- Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK; (A.N.)
| | - Minh-Son Dao
- National Institute of Information and Communications Technology (NICT), Tokyo 184-0015, Japan
| | - Amir Nassibi
- Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK; (A.N.)
| | - Danilo Mandic
- Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, UK; (A.N.)
| |
Collapse
|
7
|
Tai P, Ding P, Wang F, Gong A, Li T, Zhao L, Su L, Fu Y. Brain-computer interface paradigms and neural coding. Front Neurosci 2024; 17:1345961. [PMID: 38287988 PMCID: PMC10822902 DOI: 10.3389/fnins.2023.1345961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Accepted: 12/28/2023] [Indexed: 01/31/2024] Open
Abstract
Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.
Collapse
Affiliation(s)
- Pengrui Tai
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Su
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
8
|
Todorov D, Schnitzler A, Hirschmann J. Parkinsonian rest tremor can be distinguished from voluntary hand movements based on subthalamic and cortical activity. Clin Neurophysiol 2024; 157:146-155. [PMID: 38030516 DOI: 10.1016/j.clinph.2023.10.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 10/19/2023] [Accepted: 10/31/2023] [Indexed: 12/01/2023]
Abstract
OBJECTIVE To distinguish Parkinsonian rest tremor and different voluntary hand movements by analyzing brain activity. METHODS We re-analyzed magnetoencephalography and local field potential recordings from the subthalamic nucleus of six patients with Parkinson's disease. Data were obtained after withdrawal from dopaminergic medication (Med Off) and after administration of levodopa (Med On). Using gradient-boosted tree learning, we classified epochs as tremor, fist-clenching, forearm extension or tremor-free rest. RESULTS Subthalamic activity alone was insufficient for distinguishing the four different motor states (balanced accuracy mean: 38%, std: 7%). The combination of cortical and subthalamic features, in contrast, allowed for a much more accurate classification (balanced accuracy mean: 75%, std: 17%). Adding a single cortical area improved balanced accuracy by 17% on average, as compared to classification based on subthalamic activity alone. In most patients, the most informative cortical areas were sensorimotor cortical regions. Decoding performance was similar in Med On and Med Off. CONCLUSIONS Electrophysiological recordings allow for distinguishing several motor states, provided that cortical signals are monitored in addition to subthalamic activity. SIGNIFICANCE By combining cortical recordings, subcortical recordings and machine learning, adaptive deep brain stimulation systems might be able to detect tremor specifically and to respond adequately to several motor states.
Collapse
Affiliation(s)
- Dmitrii Todorov
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany; Centre de Recherche en Neurosciences de Lyon - Inserm U1028, 69675 Bron, France; Centre de Recerca Matemática, Campus UAB edifici C, 08193 Bellaterra, Barcelona, Spain
| | - Alfons Schnitzler
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany; Center for Movement Disorders and Neuromodulation, Department of Neurology Medical Faculty, Heinrich Heine University, 40225 Düsseldorf, Germany
| | - Jan Hirschmann
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany.
| |
Collapse
|
9
|
Xie Y, Wang K, Meng J, Yue J, Meng L, Yi W, Jung TP, Xu M, Ming D. Cross-dataset transfer learning for motor imagery signal classification via multi-task learning and pre-training. J Neural Eng 2023; 20:056037. [PMID: 37774694 DOI: 10.1088/1741-2552/acfe9c] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 09/29/2023] [Indexed: 10/01/2023]
Abstract
Objective.Deep learning (DL) models have been proven to be effective in decoding motor imagery (MI) signals in Electroencephalogram (EEG) data. However, DL models' success relies heavily on large amounts of training data, whereas EEG data collection is laborious and time-consuming. Recently, cross-dataset transfer learning has emerged as a promising approach to meet the data requirements of DL models. Nevertheless, transferring knowledge across datasets involving different MI tasks remains a significant challenge in cross-dataset transfer learning, limiting the full utilization of valuable data resources. APPROACH This study proposes a pre-training-based cross-dataset transfer learning method inspired by Hard Parameter Sharing in multi-task learning. Different datasets with distinct MI paradigms are considered as different tasks, classified with shared feature extraction layers and individual task-specific layers to allow cross-dataset classification with one unified model. Then, Pre-training and fine-tuning are employed to transfer knowledge across datasets. We also designed four fine-tuning schemes and conducted extensive experiments on them. MAIN RESULTS The results showed that compared to models without pre-training, models with pre-training achieved a maximum increase in accuracy of 7.76%. Moreover, when limited training data were available, the pre-training method significantly improved DL model's accuracy by 27.34% at most. The experiments also revealed that pre-trained models exhibit faster convergence and remarkable robustness. The training time per subject could be reduced by up to 102.83 s, and the variance of classification accuracy decreased by 75.22% at best. SIGNIFICANCE This study represents the first comprehensive investigation of the cross-dataset transfer learning method between two datasets with different MI tasks. The proposed pre-training method requires only minimal fine-tuning data when applying DL models to new MI paradigms, making MI-Brain-computer interface more practical and user-friendly.
Collapse
Affiliation(s)
- Yuting Xie
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
| | - Kun Wang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
| | - Jiayuan Meng
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, People's Republic of China
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
| | - Jin Yue
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
| | - Lin Meng
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
| | - Weibo Yi
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
- Beijing Institute of Mechanical Equipment, Beijin, People's Republic of China
| | - Tzyy-Ping Jung
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, People's Republic of China
| | - Minpeng Xu
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, People's Republic of China
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
| | - Dong Ming
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People's Republic of China
- College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, People's Republic of China
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, People's Republic of China
| |
Collapse
|
10
|
Niu L, Bin J, Wang JKS, Zhan G, Jia J, Zhang L, Gan Z, Kang X. Effect of 3D paradigm synchronous motion for SSVEP-based hybrid BCI-VR system. Med Biol Eng Comput 2023; 61:2481-2495. [PMID: 37191865 DOI: 10.1007/s11517-023-02845-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 05/05/2023] [Indexed: 05/17/2023]
Abstract
A brain-computer interface (BCI) system and virtual reality (VR) are integrated as a more interactive hybrid system (BCI-VR) that allows the user to manipulate the car. A virtual scene in the VR system that is the same as the physical environment is built, and the object's movement can be observed in the VR scene. The four-class three-dimensional (3D) paradigm is designed and moves synchronously in virtual reality. The dynamic paradigm may affect their attention according to the experimenters' feedback. Fifteen subjects in our experiment steered the car according to a specified motion trajectory. According to our online experimental result, different motion trajectories of the paradigm have various effects on the system's performance, and training can mitigate this adverse effect. Moreover, the hybrid system using frequencies between 5 and 10 Hz indicates better performance than those using lower or higher stimulation frequencies. The experiment results show a maximum average accuracy of 0.956 and a maximum information transfer rate (ITR) of 41.033 bits/min. It suggests that a hybrid system provides a high-performance way of brain-computer interaction. This research could encourage more interesting applications involving BCI and VR technologies.
Collapse
Affiliation(s)
- Lan Niu
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Jianxiong Bin
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | | | - Gege Zhan
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Jie Jia
- Department of Rehabilitation Medicine, Huashan Hospital, Fudan University, Shanghai, 200040, China
| | - Lihua Zhang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Zhongxue Gan
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Xiaoyang Kang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China.
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China.
- Yiwu Research Institute of Fudan University, Chengbei Road, Yiwu City, 322000, Zhejiang, China.
- Research Center for Intelligent Sensing, Zhejiang Lab, Hangzhou, 311100, China.
| |
Collapse
|
11
|
Hualiang L, Xupeng Y, Yuzhong L, Tingjun X, Wei T, Yali S, Qiru W, Chaolin X, Yu W, Weilin L, Long J. A novel noninvasive brain-computer interface by imagining isometric force levels. Cogn Neurodyn 2023; 17:975-983. [PMID: 37522042 PMCID: PMC10374494 DOI: 10.1007/s11571-022-09875-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2022] [Revised: 07/22/2022] [Accepted: 08/19/2022] [Indexed: 11/03/2022] Open
Abstract
Physiological circuits differ across increasing isometric force levels during unilateral contraction. Therefore, we first explored the possibility of predicting the force level based on electroencephalogram (EEG) activity recorded during a single trial of unilateral 5% or 40% of maximal isometric voluntary contraction (MVC) in right-hand grip imagination. Nine healthy subjects were involved in this study. The subjects were required to randomly perform 20 trials for each force level while imagining a right-hand grip. We proposed the use of common spatial patterns (CSPs) and coherence between EEG signals as features in a support vector machine for force level prediction. The results showed that the force levels could be predicted through single-trial EEGs while imagining the grip (mean accuracy = 81.4 ± 13.29%). Additionally, we tested the possibility of online control of a ball game using the above paradigm through unilateral grip imagination at different force levels (i.e., 5% of MVC imagination and 40% of MVC imagination for right-hand movement control). Subjects played the ball games effectively by controlling direction with our novel BCI system (n = 9, mean accuracy = 76.67 ± 9.35%). Data analysis validated the use of our BCI system in the online control of a ball game. This information may provide additional commands for the control of robots by users through combinations with other traditional brain-computer interfaces, e.g., different limb imaginations.
Collapse
Affiliation(s)
- Li Hualiang
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Ye Xupeng
- College of Information Science and Technology, and Guangdong Key Lab of Traditional Chinese Medicine Information Technology, Jinan University, Guangzhou, 510632 China
| | - Liu Yuzhong
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Xie Tingjun
- Guangdong Power Grid Co., Ltd., Guangzhou, China
| | - Tan Wei
- Guangdong Power Grid Co., Ltd., Guangzhou, China
| | - Shen Yali
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Wang Qiru
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Xiong Chaolin
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Wang Yu
- Key Laboratory of Occupational Health and Safety of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
- Electric Power Research Institute of Guangdong Power Grid Co., Ltd., Guangzhou, Guangdong China
| | - Lin Weilin
- College of Information Science and Technology, and Guangdong Key Lab of Traditional Chinese Medicine Information Technology, Jinan University, Guangzhou, 510632 China
| | - Jinyi Long
- College of Information Science and Technology, and Guangdong Key Lab of Traditional Chinese Medicine Information Technology, Jinan University, Guangzhou, 510632 China
| |
Collapse
|
12
|
He C, Chen YY, Phang CR, Stevenson C, Chen IP, Jung TP, Ko LW. Diversity and Suitability of the State-of-the-Art Wearable and Wireless EEG Systems Review. IEEE J Biomed Health Inform 2023; 27:3830-3843. [PMID: 37022001 DOI: 10.1109/jbhi.2023.3239053] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
Abstract
Wireless electroencephalography (EEG) systems have been attracting increasing attention in recent times. Both the number of articles discussing wireless EEG and their proportion relative to general EEG publications have increased over years. These trends indicate that wireless EEG systems could be more accessible to researchers and the research community has recognized the potential of wireless EEG systems. To explore the development and diverse applications of wireless EEG systems, this review highlights the trends in wearable and wireless EEG systems over the past decade and compares the specifications and research applications of the major wireless systems marketed by 16 companies. For each product, five parameters (number of channels, sampling rate, cost, battery life, and resolution) were assessed for comparison. Currently, these wearable and portable wireless EEG systems have three main application areas: consumer, clinical, and research. To address this multitude of options, the article also discussed the thought process to find a suitable device that meets personalization and use cases specificities. These investigations suggest that low-price and convenience are key factors for consumer applications, wireless EEG systems with FDA or CE-certification may be more suitable for clinical settings, and devices that provide raw EEG data with high-density channels are important for laboratory research. This article presents an overview of the current state of the wireless EEG systems specifications and possible applications and serves as a guide point as it is expected that more influential and novel research will cyclically promote the development of such EEG systems.
Collapse
|
13
|
Liang L, Zhang Q, Zhou J, Li W, Gao X. Dataset Evaluation Method and Application for Performance Testing of SSVEP-BCI Decoding Algorithm. SENSORS (BASEL, SWITZERLAND) 2023; 23:6310. [PMID: 37514603 PMCID: PMC10385518 DOI: 10.3390/s23146310] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 06/24/2023] [Accepted: 07/07/2023] [Indexed: 07/30/2023]
Abstract
Steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems have been extensively researched over the past two decades, and multiple sets of standard datasets have been published and widely used. However, there are differences in sample distribution and collection equipment across different datasets, and there is a lack of a unified evaluation method. Most new SSVEP decoding algorithms are tested based on self-collected data or offline performance verification using one or two previous datasets, which can lead to performance differences when used in actual application scenarios. To address these issues, this paper proposed a SSVEP dataset evaluation method and analyzed six datasets with frequency and phase modulation paradigms to form an SSVEP algorithm evaluation dataset system. Finally, based on the above datasets, performance tests were carried out on the four existing SSVEP decoding algorithms. The findings reveal that the performance of the same algorithm varies significantly when tested on diverse datasets. Substantial performance variations were observed among subjects, ranging from the best-performing to the worst-performing. The above results demonstrate that the SSVEP dataset evaluation method can integrate six datasets to form a SSVEP algorithm performance testing dataset system. This system can test and verify the SSVEP decoding algorithm from different perspectives such as different subjects, different environments, and different equipment, which is helpful for the research of new SSVEP decoding algorithms and has significant reference value for other BCI application fields.
Collapse
Affiliation(s)
- Liyan Liang
- China Academy of Information and Communications Technology, Beijing 100161, China
| | - Qian Zhang
- China Academy of Information and Communications Technology, Beijing 100161, China
| | - Jie Zhou
- China Academy of Information and Communications Technology, Beijing 100161, China
| | - Wenyu Li
- China Academy of Information and Communications Technology, Beijing 100161, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| |
Collapse
|
14
|
Wang P, Cao X, Zhou Y, Gong P, Yousefnezhad M, Shao W, Zhang D. A comprehensive review on motion trajectory reconstruction for EEG-based brain-computer interface. Front Neurosci 2023; 17:1086472. [PMID: 37332859 PMCID: PMC10272365 DOI: 10.3389/fnins.2023.1086472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Accepted: 05/03/2023] [Indexed: 06/20/2023] Open
Abstract
The advance in neuroscience and computer technology over the past decades have made brain-computer interface (BCI) a most promising area of neurorehabilitation and neurophysiology research. Limb motion decoding has gradually become a hot topic in the field of BCI. Decoding neural activity related to limb movement trajectory is considered to be of great help to the development of assistive and rehabilitation strategies for motor-impaired users. Although a variety of decoding methods have been proposed for limb trajectory reconstruction, there does not yet exist a review that covers the performance evaluation of these decoding methods. To alleviate this vacancy, in this paper, we evaluate EEG-based limb trajectory decoding methods regarding their advantages and disadvantages from a variety of perspectives. Specifically, we first introduce the differences in motor execution and motor imagery in limb trajectory reconstruction with different spaces (2D and 3D). Then, we discuss the limb motion trajectory reconstruction methods including experiment paradigm, EEG pre-processing, feature extraction and selection, decoding methods, and result evaluation. Finally, we expound on the open problem and future outlooks.
Collapse
Affiliation(s)
| | | | | | | | | | - Wei Shao
- *Correspondence: Wei Shao, ; Daoqiang Zhang,
| | | |
Collapse
|
15
|
Fernández-Rodríguez Á, Ron-Angevin R, Velasco-Álvarez F, Diaz-Pineda J, Letouzé T, André JM. Evaluation of Single-Trial Classification to Control a Visual ERP-BCI under a Situation Awareness Scenario. Brain Sci 2023; 13:886. [PMID: 37371365 DOI: 10.3390/brainsci13060886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Revised: 05/15/2023] [Accepted: 05/29/2023] [Indexed: 06/29/2023] Open
Abstract
An event-related potential (ERP)-based brain-computer interface (BCI) can be used to monitor a user's cognitive state during a surveillance task in a situational awareness context. The present study explores the use of an ERP-BCI for detecting new planes in an air traffic controller (ATC). Two experiments were conducted to evaluate the impact of different visual factors on target detection. Experiment 1 validated the type of stimulus used and the effect of not knowing its appearance location in an ERP-BCI scenario. Experiment 2 evaluated the effect of the size of the target stimulus appearance area and the stimulus salience in an ATC scenario. The main results demonstrate that the size of the plane appearance area had a negative impact on the detection performance and on the amplitude of the P300 component. Future studies should address this issue to improve the performance of an ATC in stimulus detection using an ERP-BCI.
Collapse
Affiliation(s)
- Álvaro Fernández-Rodríguez
- Departamento de Tecnología Electrónica, Instituto Universitario de Investigación en Telecomunicación de la Universidad de Málaga (TELMA), Universidad de Málaga, 29071 Malaga, Spain
| | - Ricardo Ron-Angevin
- Departamento de Tecnología Electrónica, Instituto Universitario de Investigación en Telecomunicación de la Universidad de Málaga (TELMA), Universidad de Málaga, 29071 Malaga, Spain
| | - Francisco Velasco-Álvarez
- Departamento de Tecnología Electrónica, Instituto Universitario de Investigación en Telecomunicación de la Universidad de Málaga (TELMA), Universidad de Málaga, 29071 Malaga, Spain
| | | | - Théodore Letouzé
- Laboratoire IMS, CNRS UMR 5218, Cognitive Team, Bordeaux INP-ENSC, 33400 Talence, France
| | - Jean-Marc André
- Laboratoire IMS, CNRS UMR 5218, Cognitive Team, Bordeaux INP-ENSC, 33400 Talence, France
| |
Collapse
|
16
|
Chang S, Wang J, Zhu Y, Wei X, Deng B, Li H, Liu C. Nonlinear dynamical modeling of neural activity using volterra series with GA-enhanced particle swarm optimization algorithm. Cogn Neurodyn 2023; 17:467-476. [PMID: 37007203 PMCID: PMC10050660 DOI: 10.1007/s11571-022-09822-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2021] [Revised: 03/23/2022] [Accepted: 05/09/2022] [Indexed: 11/03/2022] Open
Abstract
In order to improve the modeling performance of Volterra sequence for nonlinear neural activity, in this paper, a new optimization algorithm is proposed to identify Volterra sequence parameters. Algorithm combines the advantages of particle swarm optimization (PSO) and genetic algorithm (GA) improve the performance of the identification of nonlinear model parameters from rapidity and accuracy. In the modeling experiments of neural signal data generated by the neural computing model and clinical neural data set in this paper, the proposed algorithm shows its excellent potential in nonlinear neural activity modeling. Compared with PSO and GA, the algorithm can achieve less identification error, and better balance the convergence speed and identification error. Further, we explore the influence of algorithm parameters on identification efficiency, which provides possible guiding significance for parameter setting in practical application of the algorithm.
Collapse
Affiliation(s)
- Siyuan Chang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| | - Jiang Wang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| | - Yulin Zhu
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| | - Xile Wei
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| | - Bin Deng
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| | - Huiyan Li
- School of Automation and Electrical Engineering, Tianjin University of Technology and Education, Tianjin, China
| | - Chen Liu
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 30072 China
| |
Collapse
|
17
|
Xu D, Tang F, Li Y, Zhang Q, Feng X. An Analysis of Deep Learning Models in SSVEP-Based BCI: A Survey. Brain Sci 2023; 13:brainsci13030483. [PMID: 36979293 PMCID: PMC10046535 DOI: 10.3390/brainsci13030483] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/04/2023] [Accepted: 03/10/2023] [Indexed: 03/15/2023] Open
Abstract
The brain–computer interface (BCI), which provides a new way for humans to directly communicate with robots without the involvement of the peripheral nervous system, has recently attracted much attention. Among all the BCI paradigms, BCIs based on steady-state visual evoked potentials (SSVEPs) have the highest information transfer rate (ITR) and the shortest training time. Meanwhile, deep learning has provided an effective and feasible solution for solving complex classification problems in many fields, and many researchers have started to apply deep learning to classify SSVEP signals. However, the designs of deep learning models vary drastically. There are many hyper-parameters that influence the performance of the model in an unpredictable way. This study surveyed 31 deep learning models (2011–2023) that were used to classify SSVEP signals and analyzed their design aspects including model input, model structure, performance measure, etc. Most of the studies that were surveyed in this paper were published in 2021 and 2022. This survey is an up-to-date design guide for researchers who are interested in using deep learning models to classify SSVEP signals.
Collapse
Affiliation(s)
- Dongcen Xu
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; (D.X.); (F.T.); (Y.L.); (Q.Z.)
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Fengzhen Tang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; (D.X.); (F.T.); (Y.L.); (Q.Z.)
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
| | - Yiping Li
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; (D.X.); (F.T.); (Y.L.); (Q.Z.)
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
| | - Qifeng Zhang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; (D.X.); (F.T.); (Y.L.); (Q.Z.)
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
| | - Xisheng Feng
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; (D.X.); (F.T.); (Y.L.); (Q.Z.)
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
- Correspondence:
| |
Collapse
|
18
|
Parallel genetic algorithm based common spatial patterns selection on time–frequency decomposed EEG signals for motor imagery brain-computer interface. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104397] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
19
|
Identifying Thematics in a Brain-Computer Interface Research. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2023; 2023:2793211. [PMID: 36643889 PMCID: PMC9833923 DOI: 10.1155/2023/2793211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Revised: 12/21/2022] [Accepted: 12/24/2022] [Indexed: 01/05/2023]
Abstract
This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.
Collapse
|
20
|
Pan J, Chen X, Ban N, He J, Chen J, Huang H. Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation. Front Hum Neurosci 2022; 16:1077717. [PMID: 36618996 PMCID: PMC9810759 DOI: 10.3389/fnhum.2022.1077717] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2022] [Accepted: 11/23/2022] [Indexed: 12/24/2022] Open
Abstract
A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.
Collapse
Affiliation(s)
- Jiahui Pan
- *Correspondence: Jiahui Pan Haiyun Huang
| | | | | | | | | | | |
Collapse
|
21
|
Kwon J, Hwang J, Nam H, Im CH. Novel hybrid visual stimuli incorporating periodic motions into conventional flickering or pattern-reversal visual stimuli for steady-state visual evoked potential-based brain-computer interfaces. Front Neuroinform 2022; 16:997068. [PMID: 36213545 PMCID: PMC9534124 DOI: 10.3389/fninf.2022.997068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 08/30/2022] [Indexed: 11/13/2022] Open
Abstract
In this study, we proposed a new type of hybrid visual stimuli for steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), which incorporate various periodic motions into conventional flickering stimuli (FS) or pattern reversal stimuli (PRS). Furthermore, we investigated optimal periodic motions for each FS and PRS to enhance the performance of SSVEP-based BCIs. Periodic motions were implemented by changing the size of the stimulus according to four different temporal functions denoted by none, square, triangular, and sine, yielding a total of eight hybrid visual stimuli. Additionally, we developed the extended version of filter bank canonical correlation analysis (FBCCA), which is a state-of-the-art training-free classification algorithm for SSVEP-based BCIs, to enhance the classification accuracy for PRS-based hybrid visual stimuli. Twenty healthy individuals participated in the SSVEP-based BCI experiment to discriminate four visual stimuli with different frequencies. An average classification accuracy and information transfer rate (ITR) were evaluated to compare the performances of SSVEP-based BCIs for different hybrid visual stimuli. Additionally, the user's visual fatigue for each of the hybrid visual stimuli was also evaluated. As the result, for FS, the highest performances were reported when the periodic motion of the sine waveform was incorporated for all window sizes except for 3 s. For PRS, the periodic motion of the square waveform showed the highest classification accuracies for all tested window sizes. A significant statistical difference in the performance between the two best stimuli was not observed. The averaged fatigue scores were reported to be 5.3 ± 2.05 and 4.05 ± 1.28 for FS with sine-wave periodic motion and PRS with square-wave periodic motion, respectively. Consequently, our results demonstrated that FS with sine-wave periodic motion and PRS with square-wave periodic motion could effectively improve the BCI performances compared to conventional FS and PRS. In addition, thanks to its low visual fatigue, PRS with square-wave periodic motion can be regarded as the most appropriate visual stimulus for the long-term use of SSVEP-based BCIs, particularly for window sizes equal to or larger than 2 s.
Collapse
Affiliation(s)
- Jinuk Kwon
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea
- Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Jihun Hwang
- Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Hyerin Nam
- Department of Artificial Intelligence, Hanyang University, Seoul, South Korea
| | - Chang-Hwan Im
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea
- Department of Electronic Engineering, Hanyang University, Seoul, South Korea
- Department of Artificial Intelligence, Hanyang University, Seoul, South Korea
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul, South Korea
- *Correspondence: Chang-Hwan Im
| |
Collapse
|
22
|
Signal analysis and classification of a novel active brain-computer interface based on four-category sequential coding. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103857] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
23
|
A novel command generation method for SSVEP-based BCI by introducing SSVEP blocking response. Comput Biol Med 2022; 146:105521. [DOI: 10.1016/j.compbiomed.2022.105521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 04/08/2022] [Accepted: 04/11/2022] [Indexed: 11/18/2022]
|
24
|
Motor Imagery EEG Classification Based on Transfer Learning and Multi-Scale Convolution Network. MICROMACHINES 2022; 13:mi13060927. [PMID: 35744539 PMCID: PMC9228168 DOI: 10.3390/mi13060927] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/25/2022] [Revised: 05/27/2022] [Accepted: 06/09/2022] [Indexed: 11/17/2022]
Abstract
For the successful application of brain-computer interface (BCI) systems, accurate recognition of electroencephalography (EEG) signals is one of the core issues. To solve the differences in individual EEG signals and the problem of less EEG data in classification and recognition, an attention mechanism-based multi-scale convolution network was designed; the transfer learning data alignment algorithm was then introduced to explore the application of transfer learning for analyzing motor imagery EEG signals. The data set 2a of BCI Competition IV was used to verify the designed dual channel attention module migration alignment with convolution neural network (MS-AFM). Experimental results showed that the classification recognition rate improved with the addition of the alignment algorithm and adaptive adjustment in transfer learning; the average classification recognition rate of nine subjects was 86.03%.
Collapse
|
25
|
EEG emotion recognition based on enhanced SPD matrix and manifold dimensionality reduction. Comput Biol Med 2022; 146:105606. [PMID: 35588679 DOI: 10.1016/j.compbiomed.2022.105606] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2022] [Revised: 05/08/2022] [Accepted: 05/09/2022] [Indexed: 11/21/2022]
Abstract
Recently, Riemannian geometry-based pattern recognition has been widely employed to brain computer interface (BCI) researches, providing new idea for emotion recognition based on electroencephalogram (EEG) signals. Although the symmetric positive definite (SPD) matrix manifold constructed from the traditional covariance matrix contains large amount of spatial information, these methods do not perform well to classify and recognize emotions, and the high dimensionality problem still unsolved. Therefore, this paper proposes a new strategy for EEG emotion recognition utilizing Riemannian geometry with the aim of achieving better classification performance. The emotional EEG signals of 32 healthy subjects were from an open-source dataset (DEAP). The wavelet packets were first applied to extract the time-frequency features of the EEG signals, and then the features were used to construct the enhanced SPD matrix. A supervised dimensionality reduction algorithm was then designed on the Riemannian manifold to reduce the high dimensionality of the SPD matrices, gather samples of the same labels together, and separate samples of different labels as much as possible. Finally, the samples were mapped to the tangent space, and the K-nearest neighbors (KNN), Random Forest (RF) and Support Vector Machine (SVM) method were employed for classification. The proposed method achieved an average accuracy of 91.86%, 91.84% on the valence and arousal recognition tasks. Furthermore, we also obtained the superior accuracy of 86.71% on the four-class recognition task, demonstrated the superiority over state-of-the-art emotion recognition methods.
Collapse
|
26
|
Hammer J, Schirrmeister RT, Hartmann K, Marusic P, Schulze-Bonhage A, Ball T. Interpretable functional specialization emerges in deep convolutional networks trained on brain signals. J Neural Eng 2022; 19. [PMID: 35421857 DOI: 10.1088/1741-2552/ac6770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 04/14/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Functional specialization is fundamental to neural information processing. Here, we study whether and how functional specialization emerges in artificial deep convolutional neural networks (CNNs) during a brain-computer interfacing (BCI) task. APPROACH We trained CNNs to predict hand movement speed from intracranial EEG (iEEG) and delineated how units across the different CNN hidden layers learned to represent the iEEG signal. MAIN RESULTS We show that distinct, functionally interpretable neural populations emerged as a result of the training process. While some units became sensitive to either iEEG amplitude or phase, others showed bimodal behavior with significant sensitivity to both features. Pruning of highly-sensitive units resulted in a steep drop of decoding accuracy not observed for pruning of less sensitive units, highlighting the functional relevance of the amplitude- and phase-specialized populations. SIGNIFICANCE We anticipate that emergent functional specialization as uncovered here will become a key concept in research towards interpretable deep learning for neuroscience and BCI applications.
Collapse
Affiliation(s)
- Jiri Hammer
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | | | - Kay Hartmann
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | - Petr Marusic
- Department of Neurology, Motol University Hospital, V Úvalu 84, Prague, 150 06, CZECH REPUBLIC
| | - Andreas Schulze-Bonhage
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79095, GERMANY
| | - Tonio Ball
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79106, GERMANY
| |
Collapse
|
27
|
Ni Z, Xu J, Wu Y, Li M, Xu G, Xu B. Improving Cross-State and Cross-Subject Visual ERP-based BCI with Temporal Modeling and Adversarial Training. IEEE Trans Neural Syst Rehabil Eng 2022; 30:369-379. [PMID: 35133966 DOI: 10.1109/tnsre.2022.3150007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Brain-computer interface (BCI) is a useful device for people without relying on peripheral nerves and muscles. However, the performance of the event-related potential (ERP)-based BCI declines when applying it to real environments, especially in cross-state and cross-subject conditions. Here we employ temporal modeling and adversarial training to improve the visual ERP-based BCI under different mental workload states and to alleviate the problems above. The rationality of our method is that the ERP-based BCI is based on electroencephalography (EEG) signals recorded from the scalp's surface, continuously changing with time and somewhat stochastic. In this paper, we propose a hierarchical recurrent network to encode all ERP signals in each repetition at the same time and model them with a temporal manner to predict which visual event elicited an ERP. The hierarchical architecture is a simple yet effective method for organizing recurrent layers in a deep structure to model long sequence signals. Taking a cue from recent advances in adversarial training, we further applied dynamic adversarial perturbations to create adversarial examples to enhance the model performance. We conduct our experiments on one published visual ERP-based BCI task with 15 subjects and 3 different auditory workload states. The results indicate that our hierarchical method can effectively model the long sequence EEG raw data, outperform the baselines on most conditions, including cross-state and cross-subject conditions. Finally, we show how deep learning-based methods with limited EEG data can improve ERP-based BCI with adversarial training. Our code will be released at https://github.com/aispeech-lab/VisBCI.
Collapse
|
28
|
Velasco-Álvarez F, Fernández-Rodríguez Á, Vizcaíno-Martín FJ, Díaz-Estrella A, Ron-Angevin R. Brain-Computer Interface (BCI) Control of a Virtual Assistant in a Smartphone to Manage Messaging Applications. SENSORS (BASEL, SWITZERLAND) 2021; 21:3716. [PMID: 34073602 PMCID: PMC8199460 DOI: 10.3390/s21113716] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Revised: 05/21/2021] [Accepted: 05/25/2021] [Indexed: 12/13/2022]
Abstract
Brain-computer interfaces (BCI) are a type of assistive technology that uses the brain signals of users to establish a communication and control channel between them and an external device. BCI systems may be a suitable tool to restore communication skills in severely motor-disabled patients, as BCI do not rely on muscular control. The loss of communication is one of the most negative consequences reported by such patients. This paper presents a BCI system focused on the control of four mainstream messaging applications running in a smartphone: WhatsApp, Telegram, e-mail and short message service (SMS). The control of the BCI is achieved through the well-known visual P300 row-column paradigm (RCP), allowing the user to select control commands as well as spelling characters. For the control of the smartphone, the system sends synthesized voice commands that are interpreted by a virtual assistant running in the smartphone. Four tasks related to the four mentioned messaging services were tested with 15 healthy volunteers, most of whom were able to accomplish the tasks, which included sending free text e-mails to an address proposed by the subjects themselves. The online performance results obtained, as well as the results of subjective questionnaires, support the viability of the proposed system.
Collapse
Affiliation(s)
- Francisco Velasco-Álvarez
- Departamento de Tecnología Electrónica, Universidad de Málaga, 29071 Málaga, Spain; (Á.F.-R.); (F.-J.V.-M.); (A.D.-E.); (R.R.-A.)
| | | | | | | | | |
Collapse
|