Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface

A delayed matching task-based study on action sequence of motor imagery

The way people imagine greatly affects performance of brain-computer interface (BCI) based on motion imagery (MI). Action sequence is a basic unit of imitation, learning, and memory for motor behavior. Whether it influences the MI-BCI is unknown, and how to manifest this influence is difficult since the MI is a spontaneous brain activity. To investigate the influence of the action sequence, this study proposes a novel paradigm named action sequences observing and delayed matching task to use images and videos to guide people to observe, match and reinforce the memory of sequence. Seven subjects' ERPs and MI performance are analyzed under four different levels of complexities or orders of the sequence. Results demonstrated that the action sequence in terms of complexity and sequence order significantly affects the MI. The complex action in positive order obtains stronger ERD/ERS and more pronounced MI feature distributions, and yields an MI classification accuracy that is 12.3% higher than complex action in negative order (p < 0.05). In addition, the ERP amplitudes derived from the supplementary motor area show a positive correlation to the MI. This study demonstrates a new perspective of improving imagery in the MI-BCI by considering the complexity and order of the action sequences, and provides a novel index for manifesting the MI performance by ERP.

An auto-segmented multi-time window dual-scale neural network for brain-computer interfaces based on event-related potentials

Objective.Event-related potentials (ERPs) are cerebral responses to cognitive processes, also referred to as cognitive potentials. Accurately decoding ERPs can help to advance research on brain-computer interfaces (BCIs). The spatial pattern of ERP varies with time. In recent years, convolutional neural networks (CNNs) have shown promising results in electroencephalography (EEG) classification, specifically for ERP-based BCIs.Approach.This study proposes an auto-segmented multi-time window dual-scale neural network (AWDSNet). The combination of a multi-window design and a lightweight base network gives AWDSNet good performance at an acceptable cost of computing. For each individual, we create a time window set by calculating the correlation of signedR-squared values, which enables us to determine the length and number of windows automatically. The signal data are segmented based on the obtained window sets in sub-plus-global mode. Then, the multi-window data are fed into a dual-scale CNN model, where the sizes of the convolution kernels are determined by the window sizes. The use of dual-scale spatiotemporal convolution focuses on feature details while also having a large enough receptive length, and the grouping parallelism undermines the increase in the number of parameters that come with dual scaling.Main results.We evaluated the performance of AWDSNet on a public dataset and a self-collected dataset. A comparison was made with four popular methods including EEGNet, DeepConvNet, EEG-Inception, and PPNN. The experimental results show that AWDSNet has excellent classification performance with acceptable computational complexity.Significance.These results indicate that AWDSNet has great potential for applications in ERP decoding.

Task sub-type states decoding via group deep bidirectional recurrent neural network

Decoding brain states under different cognitive tasks from functional magnetic resonance imaging (fMRI) data has attracted great attention in the neuroimaging filed. However, the well-known temporal dependency in fMRI sequences has not been fully exploited in existing studies, due to the limited temporal-modeling capacity of the backbone machine learning algorithms and rigid training sample organization strategies upon which the brain decoding methods are built. To address these limitations, we propose a novel method for fine-grain brain state decoding, namely, group deep bidirectional recurrent neural network (Group-DBRNN) model. We first propose a training sample organization strategy that consists of a group-task sample generation module and a multiple-scale random fragment strategy (MRFS) module to collect training samples that contain rich task-relevant brain activity contrast (i.e., the comparison of neural activity patterns between different tasks) and maintain the temporal dependency. We then develop a novel decoding model by replacing the unidirectional RNNs that are widely used in existing brain state decoding studies with bidirectional stacked RNNs to better capture the temporal dependency, and by introducing a multi-task interaction layer (MTIL) module to effectively model the task-relevant brain activity contrast. Our experimental results on the Human Connectome Project task fMRI dataset (7 tasks consisting of 23 task sub-type states) show that the proposed model achieves an average decoding accuracy of 94.7% over the 23 fine-grain sub-type states. Meanwhile, our extensive interpretations of the intermediate features learned in the proposed model via visualizations and quantitative assessments of their discriminability and inter-subject alignment evidence that the proposed model can effectively capture the temporal dependency and task-relevant contrast.

Brain-machine interface based on deep learning to control asynchronously a lower-limb robotic exoskeleton: a case-of-study

BACKGROUND

This research focused on the development of a motor imagery (MI) based brain-machine interface (BMI) using deep learning algorithms to control a lower-limb robotic exoskeleton. The study aimed to overcome the limitations of traditional BMI approaches by leveraging the advantages of deep learning, such as automated feature extraction and transfer learning. The experimental protocol to evaluate the BMI was designed as asynchronous, allowing subjects to perform mental tasks at their own will.

METHODS

A total of five healthy able-bodied subjects were enrolled in this study to participate in a series of experimental sessions. The brain signals from two of these sessions were used to develop a generic deep learning model through transfer learning. Subsequently, this model was fine-tuned during the remaining sessions and subjected to evaluation. Three distinct deep learning approaches were compared: one that did not undergo fine-tuning, another that fine-tuned all layers of the model, and a third one that fine-tuned only the last three layers. The evaluation phase involved the exclusive closed-loop control of the exoskeleton device by the participants' neural activity using the second deep learning approach for the decoding.

RESULTS

The three deep learning approaches were assessed in comparison to an approach based on spatial features that was trained for each subject and experimental session, demonstrating their superior performance. Interestingly, the deep learning approach without fine-tuning achieved comparable performance to the features-based approach, indicating that a generic model trained on data from different individuals and previous sessions can yield similar efficacy. Among the three deep learning approaches compared, fine-tuning all layer weights demonstrated the highest performance.

CONCLUSION

This research represents an initial stride toward future calibration-free methods. Despite the efforts to diminish calibration time by leveraging data from other subjects, complete elimination proved unattainable. The study's discoveries hold notable significance for advancing calibration-free approaches, offering the promise of minimizing the need for training trials. Furthermore, the experimental evaluation protocol employed in this study aimed to replicate real-life scenarios, granting participants a higher degree of autonomy in decision-making regarding actions such as walking or stopping gait.

Graph neural network based on brain inspired forward-forward mechanism for motor imagery classification in brain-computer interfaces

Introduction

Within the development of brain-computer interface (BCI) systems, it is crucial to consider the impact of brain network dynamics and neural signal transmission mechanisms on electroencephalogram-based motor imagery (MI-EEG) tasks. However, conventional deep learning (DL) methods cannot reflect the topological relationship among electrodes, thereby hindering the effective decoding of brain activity.

Methods

Inspired by the concept of brain neuronal forward-forward (F-F) mechanism, a novel DL framework based on Graph Neural Network combined forward-forward mechanism (F-FGCN) is presented. F-FGCN framework aims to enhance EEG signal decoding performance by applying functional topological relationships and signal propagation mechanism. The fusion process involves converting the multi-channel EEG into a sequence of signals and constructing a network grounded on the Pearson correlation coeffcient, effectively representing the associations between channels. Our model initially pre-trains the Graph Convolutional Network (GCN), and fine-tunes the output layer to obtain the feature vector. Moreover, the F-F model is used for advanced feature extraction and classification.

Results and discussion

Achievement of F-FGCN is assessed on the PhysioNet dataset for a four-class categorization, compared with various classical and state-of-the-art models. The learned features of the F-FGCN substantially amplify the performance of downstream classifiers, achieving the highest accuracy of 96.11% and 82.37% at the subject and group levels, respectively. Experimental results affirm the potency of FFGCN in enhancing EEG decoding performance, thus paving the way for BCI applications.

Exploring Convolutional Neural Network Architectures for EEG Feature Extraction

The main purpose of this paper is to provide information on how to create a convolutional neural network (CNN) for extracting features from EEG signals. Our task was to understand the primary aspects of creating and fine-tuning CNNs for various application scenarios. We considered the characteristics of EEG signals, coupled with an exploration of various signal processing and data preparation techniques. These techniques include noise reduction, filtering, encoding, decoding, and dimension reduction, among others. In addition, we conduct an in-depth analysis of well-known CNN architectures, categorizing them into four distinct groups: standard implementation, recurrent convolutional, decoder architecture, and combined architecture. This paper further offers a comprehensive evaluation of these architectures, covering accuracy metrics, hyperparameters, and an appendix that contains a table outlining the parameters of commonly used CNN architectures for feature extraction from EEG signals.

Brain-computer interface paradigms and neural coding

Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.

Parkinsonian rest tremor can be distinguished from voluntary hand movements based on subthalamic and cortical activity

OBJECTIVE

To distinguish Parkinsonian rest tremor and different voluntary hand movements by analyzing brain activity.

METHODS

We re-analyzed magnetoencephalography and local field potential recordings from the subthalamic nucleus of six patients with Parkinson's disease. Data were obtained after withdrawal from dopaminergic medication (Med Off) and after administration of levodopa (Med On). Using gradient-boosted tree learning, we classified epochs as tremor, fist-clenching, forearm extension or tremor-free rest.

RESULTS

Subthalamic activity alone was insufficient for distinguishing the four different motor states (balanced accuracy mean: 38%, std: 7%). The combination of cortical and subthalamic features, in contrast, allowed for a much more accurate classification (balanced accuracy mean: 75%, std: 17%). Adding a single cortical area improved balanced accuracy by 17% on average, as compared to classification based on subthalamic activity alone. In most patients, the most informative cortical areas were sensorimotor cortical regions. Decoding performance was similar in Med On and Med Off.

CONCLUSIONS

Electrophysiological recordings allow for distinguishing several motor states, provided that cortical signals are monitored in addition to subthalamic activity.

SIGNIFICANCE

By combining cortical recordings, subcortical recordings and machine learning, adaptive deep brain stimulation systems might be able to detect tremor specifically and to respond adequately to several motor states.

Cross-dataset transfer learning for motor imagery signal classification via multi-task learning and pre-training

Objective.Deep learning (DL) models have been proven to be effective in decoding motor imagery (MI) signals in Electroencephalogram (EEG) data. However, DL models' success relies heavily on large amounts of training data, whereas EEG data collection is laborious and time-consuming. Recently, cross-dataset transfer learning has emerged as a promising approach to meet the data requirements of DL models. Nevertheless, transferring knowledge across datasets involving different MI tasks remains a significant challenge in cross-dataset transfer learning, limiting the full utilization of valuable data resources.

APPROACH

This study proposes a pre-training-based cross-dataset transfer learning method inspired by Hard Parameter Sharing in multi-task learning. Different datasets with distinct MI paradigms are considered as different tasks, classified with shared feature extraction layers and individual task-specific layers to allow cross-dataset classification with one unified model. Then, Pre-training and fine-tuning are employed to transfer knowledge across datasets. We also designed four fine-tuning schemes and conducted extensive experiments on them.

MAIN RESULTS

The results showed that compared to models without pre-training, models with pre-training achieved a maximum increase in accuracy of 7.76%. Moreover, when limited training data were available, the pre-training method significantly improved DL model's accuracy by 27.34% at most. The experiments also revealed that pre-trained models exhibit faster convergence and remarkable robustness. The training time per subject could be reduced by up to 102.83 s, and the variance of classification accuracy decreased by 75.22% at best.

SIGNIFICANCE

This study represents the first comprehensive investigation of the cross-dataset transfer learning method between two datasets with different MI tasks. The proposed pre-training method requires only minimal fine-tuning data when applying DL models to new MI paradigms, making MI-Brain-computer interface more practical and user-friendly.

Effect of 3D paradigm synchronous motion for SSVEP-based hybrid BCI-VR system

A brain-computer interface (BCI) system and virtual reality (VR) are integrated as a more interactive hybrid system (BCI-VR) that allows the user to manipulate the car. A virtual scene in the VR system that is the same as the physical environment is built, and the object's movement can be observed in the VR scene. The four-class three-dimensional (3D) paradigm is designed and moves synchronously in virtual reality. The dynamic paradigm may affect their attention according to the experimenters' feedback. Fifteen subjects in our experiment steered the car according to a specified motion trajectory. According to our online experimental result, different motion trajectories of the paradigm have various effects on the system's performance, and training can mitigate this adverse effect. Moreover, the hybrid system using frequencies between 5 and 10 Hz indicates better performance than those using lower or higher stimulation frequencies. The experiment results show a maximum average accuracy of 0.956 and a maximum information transfer rate (ITR) of 41.033 bits/min. It suggests that a hybrid system provides a high-performance way of brain-computer interaction. This research could encourage more interesting applications involving BCI and VR technologies.

A novel noninvasive brain-computer interface by imagining isometric force levels

Physiological circuits differ across increasing isometric force levels during unilateral contraction. Therefore, we first explored the possibility of predicting the force level based on electroencephalogram (EEG) activity recorded during a single trial of unilateral 5% or 40% of maximal isometric voluntary contraction (MVC) in right-hand grip imagination. Nine healthy subjects were involved in this study. The subjects were required to randomly perform 20 trials for each force level while imagining a right-hand grip. We proposed the use of common spatial patterns (CSPs) and coherence between EEG signals as features in a support vector machine for force level prediction. The results showed that the force levels could be predicted through single-trial EEGs while imagining the grip (mean accuracy = 81.4 ± 13.29%). Additionally, we tested the possibility of online control of a ball game using the above paradigm through unilateral grip imagination at different force levels (i.e., 5% of MVC imagination and 40% of MVC imagination for right-hand movement control). Subjects played the ball games effectively by controlling direction with our novel BCI system (n = 9, mean accuracy = 76.67 ± 9.35%). Data analysis validated the use of our BCI system in the online control of a ball game. This information may provide additional commands for the control of robots by users through combinations with other traditional brain-computer interfaces, e.g., different limb imaginations.

Diversity and Suitability of the State-of-the-Art Wearable and Wireless EEG Systems Review

Wireless electroencephalography (EEG) systems have been attracting increasing attention in recent times. Both the number of articles discussing wireless EEG and their proportion relative to general EEG publications have increased over years. These trends indicate that wireless EEG systems could be more accessible to researchers and the research community has recognized the potential of wireless EEG systems. To explore the development and diverse applications of wireless EEG systems, this review highlights the trends in wearable and wireless EEG systems over the past decade and compares the specifications and research applications of the major wireless systems marketed by 16 companies. For each product, five parameters (number of channels, sampling rate, cost, battery life, and resolution) were assessed for comparison. Currently, these wearable and portable wireless EEG systems have three main application areas: consumer, clinical, and research. To address this multitude of options, the article also discussed the thought process to find a suitable device that meets personalization and use cases specificities. These investigations suggest that low-price and convenience are key factors for consumer applications, wireless EEG systems with FDA or CE-certification may be more suitable for clinical settings, and devices that provide raw EEG data with high-density channels are important for laboratory research. This article presents an overview of the current state of the wireless EEG systems specifications and possible applications and serves as a guide point as it is expected that more influential and novel research will cyclically promote the development of such EEG systems.

Dataset Evaluation Method and Application for Performance Testing of SSVEP-BCI Decoding Algorithm

Steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems have been extensively researched over the past two decades, and multiple sets of standard datasets have been published and widely used. However, there are differences in sample distribution and collection equipment across different datasets, and there is a lack of a unified evaluation method. Most new SSVEP decoding algorithms are tested based on self-collected data or offline performance verification using one or two previous datasets, which can lead to performance differences when used in actual application scenarios. To address these issues, this paper proposed a SSVEP dataset evaluation method and analyzed six datasets with frequency and phase modulation paradigms to form an SSVEP algorithm evaluation dataset system. Finally, based on the above datasets, performance tests were carried out on the four existing SSVEP decoding algorithms. The findings reveal that the performance of the same algorithm varies significantly when tested on diverse datasets. Substantial performance variations were observed among subjects, ranging from the best-performing to the worst-performing. The above results demonstrate that the SSVEP dataset evaluation method can integrate six datasets to form a SSVEP algorithm performance testing dataset system. This system can test and verify the SSVEP decoding algorithm from different perspectives such as different subjects, different environments, and different equipment, which is helpful for the research of new SSVEP decoding algorithms and has significant reference value for other BCI application fields.

A comprehensive review on motion trajectory reconstruction for EEG-based brain-computer interface

The advance in neuroscience and computer technology over the past decades have made brain-computer interface (BCI) a most promising area of neurorehabilitation and neurophysiology research. Limb motion decoding has gradually become a hot topic in the field of BCI. Decoding neural activity related to limb movement trajectory is considered to be of great help to the development of assistive and rehabilitation strategies for motor-impaired users. Although a variety of decoding methods have been proposed for limb trajectory reconstruction, there does not yet exist a review that covers the performance evaluation of these decoding methods. To alleviate this vacancy, in this paper, we evaluate EEG-based limb trajectory decoding methods regarding their advantages and disadvantages from a variety of perspectives. Specifically, we first introduce the differences in motor execution and motor imagery in limb trajectory reconstruction with different spaces (2D and 3D). Then, we discuss the limb motion trajectory reconstruction methods including experiment paradigm, EEG pre-processing, feature extraction and selection, decoding methods, and result evaluation. Finally, we expound on the open problem and future outlooks.

Evaluation of Single-Trial Classification to Control a Visual ERP-BCI under a Situation Awareness Scenario

An event-related potential (ERP)-based brain-computer interface (BCI) can be used to monitor a user's cognitive state during a surveillance task in a situational awareness context. The present study explores the use of an ERP-BCI for detecting new planes in an air traffic controller (ATC). Two experiments were conducted to evaluate the impact of different visual factors on target detection. Experiment 1 validated the type of stimulus used and the effect of not knowing its appearance location in an ERP-BCI scenario. Experiment 2 evaluated the effect of the size of the target stimulus appearance area and the stimulus salience in an ATC scenario. The main results demonstrate that the size of the plane appearance area had a negative impact on the detection performance and on the amplitude of the P300 component. Future studies should address this issue to improve the performance of an ATC in stimulus detection using an ERP-BCI.

Nonlinear dynamical modeling of neural activity using volterra series with GA-enhanced particle swarm optimization algorithm

In order to improve the modeling performance of Volterra sequence for nonlinear neural activity, in this paper, a new optimization algorithm is proposed to identify Volterra sequence parameters. Algorithm combines the advantages of particle swarm optimization (PSO) and genetic algorithm (GA) improve the performance of the identification of nonlinear model parameters from rapidity and accuracy. In the modeling experiments of neural signal data generated by the neural computing model and clinical neural data set in this paper, the proposed algorithm shows its excellent potential in nonlinear neural activity modeling. Compared with PSO and GA, the algorithm can achieve less identification error, and better balance the convergence speed and identification error. Further, we explore the influence of algorithm parameters on identification efficiency, which provides possible guiding significance for parameter setting in practical application of the algorithm.

An Analysis of Deep Learning Models in SSVEP-Based BCI: A Survey

The brain–computer interface (BCI), which provides a new way for humans to directly communicate with robots without the involvement of the peripheral nervous system, has recently attracted much attention. Among all the BCI paradigms, BCIs based on steady-state visual evoked potentials (SSVEPs) have the highest information transfer rate (ITR) and the shortest training time. Meanwhile, deep learning has provided an effective and feasible solution for solving complex classification problems in many fields, and many researchers have started to apply deep learning to classify SSVEP signals. However, the designs of deep learning models vary drastically. There are many hyper-parameters that influence the performance of the model in an unpredictable way. This study surveyed 31 deep learning models (2011–2023) that were used to classify SSVEP signals and analyzed their design aspects including model input, model structure, performance measure, etc. Most of the studies that were surveyed in this paper were published in 2021 and 2022. This survey is an up-to-date design guide for researchers who are interested in using deep learning models to classify SSVEP signals.

Identifying Thematics in a Brain-Computer Interface Research

This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.

Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation

A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.

Novel hybrid visual stimuli incorporating periodic motions into conventional flickering or pattern-reversal visual stimuli for steady-state visual evoked potential-based brain-computer interfaces

In this study, we proposed a new type of hybrid visual stimuli for steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), which incorporate various periodic motions into conventional flickering stimuli (FS) or pattern reversal stimuli (PRS). Furthermore, we investigated optimal periodic motions for each FS and PRS to enhance the performance of SSVEP-based BCIs. Periodic motions were implemented by changing the size of the stimulus according to four different temporal functions denoted by none, square, triangular, and sine, yielding a total of eight hybrid visual stimuli. Additionally, we developed the extended version of filter bank canonical correlation analysis (FBCCA), which is a state-of-the-art training-free classification algorithm for SSVEP-based BCIs, to enhance the classification accuracy for PRS-based hybrid visual stimuli. Twenty healthy individuals participated in the SSVEP-based BCI experiment to discriminate four visual stimuli with different frequencies. An average classification accuracy and information transfer rate (ITR) were evaluated to compare the performances of SSVEP-based BCIs for different hybrid visual stimuli. Additionally, the user's visual fatigue for each of the hybrid visual stimuli was also evaluated. As the result, for FS, the highest performances were reported when the periodic motion of the sine waveform was incorporated for all window sizes except for 3 s. For PRS, the periodic motion of the square waveform showed the highest classification accuracies for all tested window sizes. A significant statistical difference in the performance between the two best stimuli was not observed. The averaged fatigue scores were reported to be 5.3 ± 2.05 and 4.05 ± 1.28 for FS with sine-wave periodic motion and PRS with square-wave periodic motion, respectively. Consequently, our results demonstrated that FS with sine-wave periodic motion and PRS with square-wave periodic motion could effectively improve the BCI performances compared to conventional FS and PRS. In addition, thanks to its low visual fatigue, PRS with square-wave periodic motion can be regarded as the most appropriate visual stimulus for the long-term use of SSVEP-based BCIs, particularly for window sizes equal to or larger than 2 s.

Motor Imagery EEG Classification Based on Transfer Learning and Multi-Scale Convolution Network

For the successful application of brain-computer interface (BCI) systems, accurate recognition of electroencephalography (EEG) signals is one of the core issues. To solve the differences in individual EEG signals and the problem of less EEG data in classification and recognition, an attention mechanism-based multi-scale convolution network was designed; the transfer learning data alignment algorithm was then introduced to explore the application of transfer learning for analyzing motor imagery EEG signals. The data set 2a of BCI Competition IV was used to verify the designed dual channel attention module migration alignment with convolution neural network (MS-AFM). Experimental results showed that the classification recognition rate improved with the addition of the alignment algorithm and adaptive adjustment in transfer learning; the average classification recognition rate of nine subjects was 86.03%.

EEG emotion recognition based on enhanced SPD matrix and manifold dimensionality reduction

Recently, Riemannian geometry-based pattern recognition has been widely employed to brain computer interface (BCI) researches, providing new idea for emotion recognition based on electroencephalogram (EEG) signals. Although the symmetric positive definite (SPD) matrix manifold constructed from the traditional covariance matrix contains large amount of spatial information, these methods do not perform well to classify and recognize emotions, and the high dimensionality problem still unsolved. Therefore, this paper proposes a new strategy for EEG emotion recognition utilizing Riemannian geometry with the aim of achieving better classification performance. The emotional EEG signals of 32 healthy subjects were from an open-source dataset (DEAP). The wavelet packets were first applied to extract the time-frequency features of the EEG signals, and then the features were used to construct the enhanced SPD matrix. A supervised dimensionality reduction algorithm was then designed on the Riemannian manifold to reduce the high dimensionality of the SPD matrices, gather samples of the same labels together, and separate samples of different labels as much as possible. Finally, the samples were mapped to the tangent space, and the K-nearest neighbors (KNN), Random Forest (RF) and Support Vector Machine (SVM) method were employed for classification. The proposed method achieved an average accuracy of 91.86%, 91.84% on the valence and arousal recognition tasks. Furthermore, we also obtained the superior accuracy of 86.71% on the four-class recognition task, demonstrated the superiority over state-of-the-art emotion recognition methods.

Interpretable functional specialization emerges in deep convolutional networks trained on brain signals

OBJECTIVE

Functional specialization is fundamental to neural information processing. Here, we study whether and how functional specialization emerges in artificial deep convolutional neural networks (CNNs) during a brain-computer interfacing (BCI) task.

APPROACH

We trained CNNs to predict hand movement speed from intracranial EEG (iEEG) and delineated how units across the different CNN hidden layers learned to represent the iEEG signal.

MAIN RESULTS

We show that distinct, functionally interpretable neural populations emerged as a result of the training process. While some units became sensitive to either iEEG amplitude or phase, others showed bimodal behavior with significant sensitivity to both features. Pruning of highly-sensitive units resulted in a steep drop of decoding accuracy not observed for pruning of less sensitive units, highlighting the functional relevance of the amplitude- and phase-specialized populations.

SIGNIFICANCE

We anticipate that emergent functional specialization as uncovered here will become a key concept in research towards interpretable deep learning for neuroscience and BCI applications.

Improving Cross-State and Cross-Subject Visual ERP-based BCI with Temporal Modeling and Adversarial Training

Brain-computer interface (BCI) is a useful device for people without relying on peripheral nerves and muscles. However, the performance of the event-related potential (ERP)-based BCI declines when applying it to real environments, especially in cross-state and cross-subject conditions. Here we employ temporal modeling and adversarial training to improve the visual ERP-based BCI under different mental workload states and to alleviate the problems above. The rationality of our method is that the ERP-based BCI is based on electroencephalography (EEG) signals recorded from the scalp's surface, continuously changing with time and somewhat stochastic. In this paper, we propose a hierarchical recurrent network to encode all ERP signals in each repetition at the same time and model them with a temporal manner to predict which visual event elicited an ERP. The hierarchical architecture is a simple yet effective method for organizing recurrent layers in a deep structure to model long sequence signals. Taking a cue from recent advances in adversarial training, we further applied dynamic adversarial perturbations to create adversarial examples to enhance the model performance. We conduct our experiments on one published visual ERP-based BCI task with 15 subjects and 3 different auditory workload states. The results indicate that our hierarchical method can effectively model the long sequence EEG raw data, outperform the baselines on most conditions, including cross-state and cross-subject conditions. Finally, we show how deep learning-based methods with limited EEG data can improve ERP-based BCI with adversarial training. Our code will be released at https://github.com/aispeech-lab/VisBCI.

Brain-Computer Interface (BCI) Control of a Virtual Assistant in a Smartphone to Manage Messaging Applications

Brain-computer interfaces (BCI) are a type of assistive technology that uses the brain signals of users to establish a communication and control channel between them and an external device. BCI systems may be a suitable tool to restore communication skills in severely motor-disabled patients, as BCI do not rely on muscular control. The loss of communication is one of the most negative consequences reported by such patients. This paper presents a BCI system focused on the control of four mainstream messaging applications running in a smartphone: WhatsApp, Telegram, e-mail and short message service (SMS). The control of the BCI is achieved through the well-known visual P300 row-column paradigm (RCP), allowing the user to select control commands as well as spelling characters. For the control of the smartphone, the system sends synthesized voice commands that are interpreted by a virtual assistant running in the smartphone. Four tasks related to the four mentioned messaging services were tested with 15 healthy volunteers, most of whom were able to accomplish the tasks, which included sending free text e-mails to an address proposed by the subjects themselves. The online performance results obtained, as well as the results of subjective questionnaires, support the viability of the proposed system.