A novel hybrid deep learning scheme for four-class motor imagery classification

Deep learning classification of EEG-based BCI monitoring of the attempted arm and hand movements

OBJECTIVES

The primary objective of this research is to improve the average classification performance for specific movements in patients with cervical spinal cord injury (SCI).

METHODS

The study utilizes a low-frequency multi-class electroencephalography (EEG) dataset from Graz University of Technology. The research combines convolutional neural network (CNN) and long-short-term memory (LSTM) architectures to uncover neural correlations between temporal and spatial aspects of the EEG signals associated with attempted arm and hand movements. To achieve this, three different methods are used to select relevant features, and the proposed model's robustness against variations in the data is validated using 10-fold cross-validation (CV). The research also investigates subject-specific adaptation in an online paradigm, extending movement classification proof-of-concept.

RESULTS

The combined CNN-LSTM model, enhanced by three feature selection methods, demonstrates robustness with a mean accuracy of 75.75 % and low standard deviation (+/- 0.74 %) in 10-fold cross-validation, confirming its reliability.

CONCLUSIONS

In summary, this research aims to make valuable contributions to the field of neuro-technology by developing EEG-controlled assistive devices using a generalized brain-computer interface (BCI) and deep learning (DL) framework. The focus is on capturing high-level spatiotemporal features and latent dependencies to enhance the performance and usability of EEG-based assistive technologies.

Attention Induced Dual Convolutional-Capsule Network (AIDC-CN): A deep learning framework for motor imagery classification

In recent times, Electroencephalography (EEG)-based motor imagery (MI) decoding has garnered significant attention due to its extensive applicability in healthcare, including areas such as assistive robotics and rehabilitation engineering. Nevertheless, the decoding of EEG signals presents considerable challenges owing to their inherent complexity, non-stationary characteristics, and low signal-to-noise ratio. Notably, deep learning-based classifiers have emerged as a prominent focus for addressing the EEG signal decoding process. This study introduces a novel deep learning classifier named the Attention Induced Dual Convolutional-Capsule Network (AIDC-CN) with the specific aim of accurately categorizing various motor imagination class labels. To enhance the classifier's performance, a dual feature extraction approach leveraging spectrogram and brain connectivity networks has been employed, diversifying the feature set in the classification task. The main highlights of the proposed AIDC-CN classifier includes the introduction of a dual convolution layer to handle the brain connectivity and spectrogram features, addition of a novel self-attention module (SAM) to accentuate the relevant parts of the convolved spectrogram features, introduction of a new cross-attention module (CAM) to refine the outputs obtained from the dual convolution layers and incorporation of a Gaussian Error Linear Unit (GELU) based dynamic routing algorithm to strengthen the coupling among the primary and secondary capsule layers. Performance analysis undertaken on four public data sets depict the superior performance of the proposed model with respect to the state-of-the-art techniques. The code for this model is available at https://github.com/RiteshSurChowdhury/AIDC-CN.

A hybrid capsule attention-based convolutional bi-GRU method for multi-class mental task classification based brain-computer Interface

Electroencephalography analysis is critical for brain computer interface research. The primary goal of brain-computer interface is to establish communication between impaired people and others via brain signals. The classification of multi-level mental activities using the brain-computer interface has recently become more difficult, which affects the accuracy of the classification. However, several deep learning-based techniques have attempted to identify mental tasks using multidimensional data. The hybrid capsule attention-based convolutional bidirectional gated recurrent unit model was introduced in this study as a hybrid deep learning technique for multi-class mental task categorization. Initially, the obtained electroencephalography data is pre-processed with a digital low-pass Butterworth filter and a discrete wavelet transform to remove disturbances. The spectrally adaptive common spatial pattern is used to extract characteristics from pre-processed electroencephalography data. The retrieved features were then loaded into the suggested classification model, which was used to extract the features deeply and classify the mental tasks. To improve classification results, the model's parameters are fine-tuned using a dung beetle optimization approach. Finally, the proposed classifier is assessed for several types of mental task classification using the provided dataset. The simulation results are compared with the existing state-of-the-art techniques in terms of accuracy, precision, recall, etc. The accuracy obtained using the proposed approach is 97.87%, which is higher than that of the other existing methods.

A 4-DOF Exosuit Using a Hybrid EEG-Based Control Approach for Upper-Limb Rehabilitation

Rehabilitation devices, such as traditional rigid exoskeletons or exosuits, have been widely used to rehabilitate upper limb function post-stroke. In this paper, we have developed an exosuit with four degrees of freedom to enable users to involve more joints in the rehabilitation process. Additionally, a hybrid electroencephalogram-based (EEG-based) control approach has been developed to promote active user engagement and provide more control commands.The hybrid EEG-based control approach includes steady-state visual evoked potential (SSVEP) paradigm and motor imagery (MI) paradigm. Firstly, the rehabilitation movement was selected by SSVEP paradigm, and the multivariate variational mode decomposition (MVMD) and canonical correlation analysis (CCA) method was used for SSVEP EEG recognition; then, the motion intention was obtained by MI paradigm, and the convolutional neural network (CNN) and long short-term memory network (LSTM) were used to build a CNN-LSTM model for MI EEG recognition; finally, the recognition results were translated into control commands of Bowden cables to achieve multi-degree-of-freedom rehabilitation.Experimental results show that the average classification accuracy of the CNN-LSTM model reaches to 90.07% ± 2.23%, and the overall accuracy of the hybrid EEG-based control approach reaches to 85.26% ± 1.95%. The twelve subjects involved in the usability assessment demonstrated an average system usability scale (SUS) score of 81.25 ± 5.82. Additionally, four participants who underwent a 35-day rehabilitation training demonstrated an average 10.33% increase in range of motion (ROM) across 4 joints, along with a 11.35% increase in the average electromyography (EMG) amplitude of the primary muscle involved.The exosuit demonstrates good accuracy in control, exhibits favorable usability, and shows certain efficacy in multi-joint rehabilitation. Our study has taken into account the neuroplastic principles, aiming to achieve active user engagement while introducing additional degrees of freedom, offering novel ideas and methods for potential brain-computer interface (BCI)-based rehabilitation strategies and hardware development.Clinical impact: Our study presents an exosuit with four degrees of freedom for stroke rehabilitation, enabling multi-joint movement and improved motor recovery. The hybrid EEG-based control approach enhances active user engagement, offering a promising strategy for more effective and user-driven rehabilitation, potentially improving clinical outcomes.Clinical and Translational Impact Statement: By developing an exosuit and a hybrid EEG-based control approach, this study enhances stroke rehabilitation through better user engagement and multi-joint capabilities. These innovations consider neuroplasticity principles, integrating rehabilitation theory with rehabilitation device.

EEG-based motor imagery channel selection and classification using hybrid optimization and two-tier deep learning

Brain-computer interface (BCI) technology holds promise for individuals with profound motor impairments, offering the potential for communication and control. Motor imagery (MI)-based BCI systems are particularly relevant in this context. Despite their potential, achieving accurate and robust classification of MI tasks using electroencephalography (EEG) data remains a significant challenge. In this paper, we employed the Minimum Redundancy Maximum Relevance (MRMR) algorithm to optimize channel selection. Furthermore, we introduced a hybrid optimization approach that combines the War Strategy Optimization (WSO) and Chimp Optimization Algorithm (ChOA). This hybridization significantly enhances the classification model's overall performance and adaptability. A two-tier deep learning architecture is proposed for classification, consisting of a Convolutional Neural Network (CNN) and a modified Deep Neural Network (M-DNN). The CNN focuses on capturing temporal correlations within EEG data, while the M-DNN is designed to extract high-level spatial characteristics from selected EEG channels. Integrating optimal channel selection, hybrid optimization, and the two-tier deep learning methodology in our BCI framework presents an enhanced approach for precise and effective BCI control. Our model got 95.06% accuracy with high precision. This advancement has the potential to significantly impact neurorehabilitation and assistive technology applications, facilitating improved communication and control for individuals with motor impairments.

Improving classification performance of motor imagery BCI through EEG data augmentation with conditional generative adversarial networks

In brain-computer interface (BCI), building accurate electroencephalogram (EEG) classifiers for specific mental tasks is critical for BCI performance. The classifiers are developed by machine learning (ML) and deep learning (DL) techniques, requiring a large dataset for training to build reliable and accurate models. However, collecting large enough EEG datasets is difficult due to intra-/inter-subject variabilities and experimental costs. This leads to the data scarcity problem, which causes overfitting issues to training samples, resulting in reducing generalization performance. To solve the EEG data scarcity problem and improve the performance of the EEG classifiers, we propose a novel EEG data augmentation (DA) framework using conditional generative adversarial networks (cGANs). An experimental study is implemented with two public EEG datasets, including motor imagery (MI) tasks (BCI competition IV IIa and III IVa), to validate the effectiveness of the proposed EEG DA method for the EEG classifiers. To evaluate the proposed cGAN-based DA method, we tested eight EEG classifiers for the experiment, including traditional MLs and state-of-the-art DLs with three existing EEG DA methods. Experimental results showed that most DA methods with proper DA proportion in the training dataset had higher classification performances than without DA. Moreover, applying the proposed DA method showed superior classification performance improvement than the other DA methods. This shows that the proposed method is a promising EEG DA method for enhancing the performances of the EEG classifiers in MI-based BCIs.

An end-to-end multi-task motor imagery EEG classification neural network based on dynamic fusion of spectral-temporal features

Electroencephalograph (EEG) brain-computer interfaces (BCI) have potential to provide new paradigms for controlling computers and devices. The accuracy of brain pattern classification in EEG BCI is directly affected by the quality of features extracted from EEG signals. Currently, feature extraction heavily relies on prior knowledge to engineer features (for example from specific frequency bands); therefore, better extraction of EEG features is an important research direction. In this work, we propose an end-to-end deep neural network that automatically finds and combines features for motor imagery (MI) based EEG BCI with 4 or more imagery classes (multi-task). First, spectral domain features of EEG signals are learned by compact convolutional neural network (CCNN) layers. Then, gated recurrent unit (GRU) neural network layers automatically learn temporal patterns. Lastly, an attention mechanism dynamically combines (across EEG channels) the extracted spectral-temporal features, reducing redundancy. We test our method using BCI Competition IV-2a and a data set we collected. The average classification accuracy on 4-class BCI Competition IV-2a was 85.1 % ± 6.19 %, comparable to recent work in the field and showing low variability among participants; average classification accuracy on our 6-class data was 64.4 % ± 8.35 %. Our dynamic fusion of spectral-temporal features is end-to-end and has relatively few network parameters, and the experimental results show its effectiveness and potential.

A novel method for modeling effective connections between brain regions based on EEG signals and graph neural networks for motor imagery detection

Classified as biomedical signal processing, cerebral signal processing plays a key role in human-computer interaction (HCI) and medical diagnosis. The motor imagery (MI) problem is an important research area in this field. Accurate solutions to this problem will greatly affect real-world applications. Most of the proposed methods are based on raw signal processing techniques. Known as prior knowledge, the structural-functional information and interregional connections can improve signal processing accuracy. It is possible to correctly perceive the generated signals by considering the brain structure (i.e. anatomical units), the source of signals, and the structural-functional dependence of different brain regions (i.e. effective connection) that are the semantic generators of signals. This study employed electroencephalograph (EEG) signals based on the activity of brain regions (cortex) and effective connections between brain regions based on dynamic causal modeling to solve the MI problem. EEG signals, as well as effective connections between brain regions to improve the interpretability of MI action, were fed into the architecture of Graph Convolutional Neural Network (GCN). The proposed model allowed GCN to extract more discriminative features. The results indicated that the proposed method was successful in developing a model with a MI detection accuracy of 93.73%.

GCNs-Net: A Graph Convolutional Neural Network Approach for Decoding Time-Resolved EEG Motor Imagery Signals

Toward the development of effective and efficient brain-computer interface (BCI) systems, precise decoding of brain activity measured by an electroencephalogram (EEG) is highly demanded. Traditional works classify EEG signals without considering the topological relationship among electrodes. However, neuroscience research has increasingly emphasized network patterns of brain dynamics. Thus, the Euclidean structure of electrodes might not adequately reflect the interaction between signals. To fill the gap, a novel deep learning (DL) framework based on the graph convolutional neural networks (GCNs) is presented to enhance the decoding performance of raw EEG signals during different types of motor imagery (MI) tasks while cooperating with the functional topological relationship of electrodes. Based on the absolute Pearson's matrix of overall signals, the graph Laplacian of EEG electrodes is built up. The GCNs-Net constructed by graph convolutional layers learns the generalized features. The followed pooling layers reduce dimensionality, and the fully-connected (FC) softmax layer derives the final prediction. The introduced approach has been shown to converge for both personalized and groupwise predictions. It has achieved the highest averaged accuracy, 93.06% and 88.57% (PhysioNet dataset), 96.24% and 80.89% (high gamma dataset), at the subject and group level, respectively, compared with existing studies, which suggests adaptability and robustness to individual variability. Moreover, the performance is stably reproducible among repetitive experiments for cross-validation. The excellent performance of our method has shown that it is an important step toward better BCI approaches. To conclude, the GCNs-Net filters EEG signals based on the functional topological relationship, which manages to decode relevant features for brain MI. A DL library for EEG task classification including the code for this study is open source at https://github.com/SuperBruceJia/ EEG-DL for scientific research.

BrainGridNet: A two-branch depthwise CNN for decoding EEG-based multi-class motor imagery

Brain-computer interfaces (BCIs) based on motor imagery (MI) enable the disabled to interact with the world through brain signals. To meet demands of real-time, stable, and diverse interactions, it is crucial to develop lightweight networks that can accurately and reliably decode multi-class MI tasks. In this paper, we introduce BrainGridNet, a convolutional neural network (CNN) framework that integrates two intersecting depthwise CNN branches with 3D electroencephalography (EEG) data to decode a five-class MI task. The BrainGridNet attains competitive results in both the time and frequency domains, with superior performance in the frequency domain. As a result, an accuracy of 80.26 percent and a kappa value of 0.753 are achieved by BrainGridNet, surpassing the state-of-the-art (SOTA) model. Additionally, BrainGridNet shows optimal computational efficiency, excels in decoding the most challenging subject, and maintains robust accuracy despite the random loss of 16 electrode signals. Finally, the visualizations demonstrate that BrainGridNet learns discriminative features and identifies critical brain regions and frequency bands corresponding to each MI class. The convergence of BrainGridNet's strong feature extraction capability, high decoding accuracy, steady decoding efficacy, and low computational costs renders it an appealing choice for facilitating the development of BCIs.

An EEG motor imagery dataset for brain computer interface in acute stroke patients

The brain-computer interface (BCI) is a technology that involves direct communication with parts of the brain and has evolved rapidly in recent years; it has begun to be used in clinical practice, such as for patient rehabilitation. Patient electroencephalography (EEG) datasets are critical for algorithm optimization and clinical applications of BCIs but are rare at present. We collected data from 50 acute stroke patients with wireless portable saline EEG devices during the performance of two tasks: 1) imagining right-handed movements and 2) imagining left-handed movements. The dataset consists of four types of data: 1) the motor imagery instructions, 2) raw recording data, 3) pre-processed data after removing artefacts and other manipulations, and 4) patient characteristics. This is the first open dataset to address left- and right-handed motor imagery in acute stroke patients. We believe that the dataset will be very helpful for analysing brain activation and designing decoding methods that are more applicable for acute stroke patients, which will greatly facilitate research in the field of motor imagery-BCI.

Spatial Visual Imagery (SVI)-Based Electroencephalograph Discrimination for Natural CAD Manipulation

With the increasing demand for natural interactions, people have realized that an intuitive Computer-Aided Design (CAD) interaction mode can reduce the complexity of CAD operation and improve the design experience. Although interaction modes like gaze and gesture are compatible with some complex CAD manipulations, they still require people to express their design intentions physically. The brain contains design intentions implicitly and controls the corresponding body parts that execute the task. Therefore, building an end-to-end channel between the brain and computer as an auxiliary mode for CAD manipulation will allow people to send design intentions mentally and make their interaction more intuitive. This work focuses on the 1-D translation scene and studies a spatial visual imagery (SVI) paradigm to provide theoretical support for building an electroencephalograph (EEG)-based brain-computer interface (BCI) for CAD manipulation. Based on the analysis of three spatial EEG features related to SVI (e.g., common spatial patterns, cross-correlation, and coherence), a multi-feature fusion-based discrimination model was built for SVI. The average accuracy of the intent discrimination of 10 subjects was 86%, and the highest accuracy was 93%. The method proposed was verified to be feasible for discriminating the intentions of CAD object translation with good classification performance. This work further proves the potential of BCI in natural CAD manipulation.

An in-depth survey on Deep Learning-based Motor Imagery Electroencephalogram (EEG) classification

Electroencephalogram (EEG)-based Brain-Computer Interfaces (BCIs) build a communication path between human brain and external devices. Among EEG-based BCI paradigms, the most commonly used one is motor imagery (MI). As a hot research topic, MI EEG-based BCI has largely contributed to medical fields and smart home industry. However, because of the low signal-to-noise ratio (SNR) and the non-stationary characteristic of EEG data, it is difficult to correctly classify different types of MI-EEG signals. Recently, the advances in Deep Learning (DL) significantly facilitate the development of MI EEG-based BCIs. In this paper, we provide a systematic survey of DL-based MI-EEG classification methods. Specifically, we first comprehensively discuss several important aspects of DL-based MI-EEG classification, covering input formulations, network architectures, public datasets, etc. Then, we summarize problems in model performance comparison and give guidelines to future studies for fair performance comparison. Next, we fairly evaluate the representative DL-based models using source code released by the authors and meticulously analyse the evaluation results. By performing ablation study on the network architecture, we found that (1) effective feature fusion is indispensable for multi-stream CNN-based models. (2) LSTM should be combined with spatial feature extraction techniques to obtain good classification performance. (3) the use of dropout contributes little to improving the model performance, and that (4) adding fully connected layers to the models significantly increases their parameters but it might not improve their performance. Finally, we raise several open issues in MI-EEG classification and provide possible future research directions.

Hierarchical Transformer for Motor Imagery-Based Brain Computer Interface

In this paper, we propose a novel transformer-based classification algorithm for the brain computer interface (BCI) using a motor imagery (MI) electroencephalogram (EEG) signal. To design the MI classification algorithm, we apply an up-to-date deep learning model, the transformer, that has revolutionized the natural language processing (NLP) and successfully widened its application to many other domains such as the computer vision. Within a long MI trial spanning a few seconds, the classification algorithm should give more attention to the time periods during which the intended motor task is imagined by the subject without any artifact. To achieve this goal, we propose a hierarchical transformer architecture that consists of a high-level transformer (HLT) and a low-level transformer (LLT). We break down a long MI trial into a number of short-term intervals. The LLT extracts a feature from each short-term interval, and the HLT pays more attention to the features from more relevant short-term intervals by using the self-attention mechanism of the transformer. We have done extensive tests of the proposed scheme on four open MI datasets, and shown that the proposed hierarchical transformer excels in both the subject-dependent and subject-independent tests.

Feature learning framework based on EEG graph self-attention networks for motor imagery BCI systems

Learning distinguishable features from raw EEG signals is crucial for accurate classification of motor imagery (MI) tasks. To incorporate spatial relationships between EEG sources, we developed a feature set based on an EEG graph. In this graph, EEG channels represent the nodes, with power spectral density (PSD) features defining their properties, and the edges preserving the spatial information. We designed an EEG based graph self-attention network (EGSAN) to learn low-dimensional embedding vector for EEG graph, which can be used as distinguishable features for motor imagery task classification. We evaluated our EGSAN model on two publicly available MI EEG datasets, each containing different types of motor imagery tasks. Our experiments demonstrate that our proposed model effectively extracts distinguishable features from EEG graphs, achieving significantly higher classification accuracies than existing state-of-the-art methods.

MBGA-Net: A multi-branch graph adaptive network for individualized motor imagery EEG classification

BACKGROUND AND OBJECTIVE

The development of deep learning has led to significant improvements in the decoding accuracy of Motor Imagery (MI) EEG signal classification. However, current models are inadequate in ensuring high levels of classification accuracy for an individual. Since MI EEG data is primarily used in medical rehabilitation and intelligent control, it is crucial to ensure that each individual's EEG signal is recognized with precision.

METHODS

We propose a multi-branch graph adaptive network (MBGA-Net), which matches each individual EEG signal with a suitable time-frequency domain processing method based on spatio-temporal domain features. We then feed the signal into the relevant model branch using an adaptive technique. Through an enhanced attention mechanism and deep convolutional method with residual connectivity, each model branch more effectively harvests the features of the related format data.

RESULTS

We validate the proposed model using the BCI Competition IV dataset 2a and dataset 2b. On dataset 2a, the average accuracy and kappa values are 87.49% and 0.83, respectively. The standard deviation of individual kappa values is only 0.08. For dataset 2b, the average classification accuracies obtained by feeding the data into the three branches of MBGA-Net are 85.71%, 85.83%, and 86.99%, respectively.

CONCLUSIONS

The experimental results demonstrate that MBGA-Net could effectively perform the classification task of motor imagery EEG signals, and it exhibits strong generalization performance. The proposed adaptive matching technique enhances the classification accuracy of each individual, which is beneficial for the practical application of EEG classification.

Local and global convolutional transformer-based motor imagery EEG classification

Transformer, a deep learning model with the self-attention mechanism, combined with the convolution neural network (CNN) has been successfully applied for decoding electroencephalogram (EEG) signals in Motor Imagery (MI) Brain-Computer Interface (BCI). However, the extremely non-linear, nonstationary characteristics of the EEG signals limits the effectiveness and efficiency of the deep learning methods. In addition, the variety of subjects and the experimental sessions impact the model adaptability. In this study, we propose a local and global convolutional transformer-based approach for MI-EEG classification. The local transformer encoder is combined to dynamically extract temporal features and make up for the shortcomings of the CNN model. The spatial features from all channels and the difference in hemispheres are obtained to improve the robustness of the model. To acquire adequate temporal-spatial feature representations, we combine the global transformer encoder and Densely Connected Network to improve the information flow and reuse. To validate the performance of the proposed model, three scenarios including within-session, cross-session and two-session are designed. In the experiments, the proposed method achieves up to 1.46%, 7.49% and 7.46% accuracy improvement respectively in the three scenarios for the public Korean dataset compared with current state-of-the-art models. For the BCI competition IV 2a dataset, the proposed model also achieves a 2.12% and 2.21% improvement for the cross-session and two-session scenarios respectively. The results confirm that the proposed approach can effectively extract much richer set of MI features from the EEG signals and improve the performance in the BCI applications.

Deep learning-based framework for real-time upper limb motion intention classification using combined bio-signals

This research study proposes a unique framework that takes input from a surface electromyogram (sEMG) and functional near-infrared spectroscopy (fNIRS) bio-signals. These signals are trained using convolutional neural networks (CNN). The framework entails a real-time neuro-machine interface to decode the human intention of upper limb motions. The bio-signals from the two modalities are recorded for eight movements simultaneously for prosthetic arm functions focusing on trans-humeral amputees. The fNIRS signals are acquired from the human motor cortex, while sEMG is recorded from the human bicep muscles. The selected classification and command generation features are the peak, minimum, and mean ΔHbO and ΔHbR values within a 2-s moving window. In the case of sEMG, wavelength, peak, and mean were extracted with a 150-ms moving window. It was found that this scheme generates eight motions with an enhanced average accuracy of 94.5%. The obtained results validate the adopted research methodology and potential for future real-time neural-machine interfaces to control prosthetic arms.

Subject-Independent EEG Classification of Motor Imagery Based on Dual-Branch Feature Fusion

A brain computer interface (BCI) system helps people with motor dysfunction interact with the external environment. With the advancement of technology, BCI systems have been applied in practice, but their practicability and usability are still greatly challenged. A large amount of calibration time is often required before BCI systems are used, which can consume the patient's energy and easily lead to anxiety. This paper proposes a novel motion-assisted method based on a novel dual-branch multiscale auto encoder network (MSAENet) to decode human brain motion imagery intentions, while introducing a central loss function to compensate for the shortcomings of traditional classifiers that only consider inter-class differences and ignore intra-class coupling. The effectiveness of the method is validated on three datasets, namely BCIIV2a, SMR-BCI and OpenBMI, to achieve zero calibration of the MI-BCI system. The results show that our proposed network displays good results on all three datasets. In the case of subject-independence, the MSAENet outperformed the other four comparison methods on the BCIIV2a and SMR-BCI datasets, while achieving F1_score values as high as 69.34% on the OpenBMI dataset. Our method maintains better classification accuracy with a small number of parameters and short prediction times, and the method achieves zero calibration of the MI-BCI system.

Overlapping filter bank convolutional neural network for multisubject multicategory motor imagery brain-computer interface

BACKGROUND

Motor imagery brain-computer interfaces (BCIs) is a classic and potential BCI technology achieving brain computer integration. In motor imagery BCI, the operational frequency band of the EEG greatly affects the performance of motor imagery EEG recognition model. However, as most algorithms used a broad frequency band, the discrimination from multiple sub-bands were not fully utilized. Thus, using convolutional neural network (CNNs) to extract discriminative features from EEG signals of different frequency components is a promising method in multisubject EEG recognition.

METHODS

This paper presents a novel overlapping filter bank CNN to incorporate discriminative information from multiple frequency components in multisubject motor imagery recognition. Specifically, two overlapping filter banks with fixed low-cut frequency or sliding low-cut frequency are employed to obtain multiple frequency component representations of EEG signals. Then, multiple CNN models are trained separately. Finally, the output probabilities of multiple CNN models are integrated to determine the predicted EEG label.

RESULTS

Experiments were conducted based on four popular CNN backbone models and three public datasets. And the results showed that the overlapping filter bank CNN was efficient and universal in improving multisubject motor imagery BCI performance. Specifically, compared with the original backbone model, the proposed method can improve the average accuracy by 3.69 percentage points, F1 score by 0.04, and AUC by 0.03. In addition, the proposed method performed best among the comparison with the state-of-the-art methods.

CONCLUSION

The proposed overlapping filter bank CNN framework with fixed low-cut frequency is an efficient and universal method to improve the performance of multisubject motor imagery BCI.

Accurate multi-objective prediction of CO2 emission performance indexes and industrial structure optimization using multihead attention-based convolutional neural network

The establishment of specific targets for the global carbon peaking and neutrality raises urgent requirements for prediction of CO₂ emission performance indexes (CEPIs) and industrial structure optimization. However, accurate multi-objective prediction of CEPIs is still a knotty problem. In the present study, multihead attention-based convolutional neural network (MHA-CNN) model was proposed for accurate prediction of 4 CEPIs and further provided the rational suggestions for further industrial structure optimization. The proposed MHA-CNN model introduces deep learning mechanism with efficient resolution strategies for training model overfitting, feature extraction, and self-supervised learning to acquire the adaptability for CEPIs. Multihead attention (MHA) mechanism plays important roles in influence weight interpretation of variables to facilitate the prediction performance of CNN on CEPIs. The MHA-CNN model presented its overwhelmingly superior performance to CNN model and long short-term memory (LSTM) model, two frequently-used models, in multi-objective prediction of CEPIs using 8 influence variables, which highlighted advantages of MHA module in multi-dimensional feature extraction. Additionally, contributions of influence variables to CEPIs based on MHA analyses presented relatively high consistency with the geographical distribution analyses, indicating the excellent capacity of the MHA module in variable weights identification and contribution dissection. Based on the more accurate prediction results by MHA-CNN than those by CNN and LSTM model, the increase in the tertiary industry and the decreases in the first and secondary industries are conducive to improvement of total-factor carbon emission efficiency and further enhancement of effective energy utilization in regions with inefficient carbon emissions. This study provides insights towards the critical roles of the proposed MHA-CNN model in accurate multi-objective prediction of CEPIs and further industrial structure optimization for improvement of total-factor carbon emission efficiency.

An Evaluation of Hybrid Deep Learning Models for Classifying Multiple Lower Limb Actions

Brain-computer Interfaces (BCIs) interpret electroencephalography (EEG) signals and translate them into control commands for operating external devices. The motor imagery (MI) paradigm is popular in this context. Recent research has demonstrated that deep learning models, such as convolutional neural network (CNN) and long short-term memory (LSTM), are successful in a wide range of classification applications. This is because CNN has the property of spatial invariance, and LSTM can capture temporal associations among features. A combination of CNN and LSTM could enhance the classification performance of EEG signals due to the complementation of their strengths. Such a combination has been applied to MI classification based on EEG. However, most studies focused on either the upper limbs or treated both lower limbs as a single class, with only limited research performed on separate lower limbs. We, therefore, explored hybrid models (different combinations of CNN and LSTM) and evaluated them in the case of individual lower limbs. In addition, we classified multiple actions: MI, real movements and movement observations using four typical hybrid models and aimed to identify which model was the most suitable. The comparison results demonstrated that no model was significantly better than the others in terms of classification accuracy, but all of them were better than the chance level. Our study informs the possibility of the use of multiple actions in BCI systems and provides useful information for further research into the classification of separate lower limb actions.

The effects of layer-wise relevance propagation-based feature selection for EEG classification: a comparative study on multiple datasets

Introduction

The brain-computer interface (BCI) allows individuals to control external devices using their neural signals. One popular BCI paradigm is motor imagery (MI), which involves imagining movements to induce neural signals that can be decoded to control devices according to the user's intention. Electroencephalography (EEG) is frequently used for acquiring neural signals from the brain in the fields of MI-BCI due to its non-invasiveness and high temporal resolution. However, EEG signals can be affected by noise and artifacts, and patterns of EEG signals vary across different subjects. Therefore, selecting the most informative features is one of the essential processes to enhance classification performance in MI-BCI.

Methods

In this study, we design a layer-wise relevance propagation (LRP)-based feature selection method which can be easily integrated into deep learning (DL)-based models. We assess its effectiveness for reliable class-discriminative EEG feature selection on two different publicly available EEG datasets with various DL-based backbone models in the subject-dependent scenario.

Results and discussion

The results show that LRP-based feature selection enhances the performance for MI classification on both datasets for all DL-based backbone models. Based on our analysis, we believe that it can broad its capability to different research domains.

Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance

Introduction

In brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.

Methods

We evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.

Results

Our results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.

Discussion

DL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.

Classification of Motor Imagery EEG Signals Based on Data Augmentation and Convolutional Neural Networks

In brain-computer interface (BCI) systems, motor imagery electroencephalography (MI-EEG) signals are commonly used to detect participant intent. Many factors, including low signal-to-noise ratios and few high-quality samples, make MI classification difficult. In order for BCI systems to function, MI-EEG signals must be studied. In pattern recognition and other fields, deep learning approaches have recently been successfully applied. In contrast, few effective deep learning algorithms have been applied to BCI systems, especially MI-based systems. In this paper, we address these problems from two aspects based on the characteristics of EEG signals: first, we proposed a combined time-frequency domain data enhancement method. This method guarantees that the size of the training data is effectively increased while maintaining the intrinsic composition of the data. Second, our design consists of a parallel CNN that takes both raw EEG images and images transformed through continuous wavelet transform (CWT) as inputs. We conducted classification experiments on a public data set to verify the effectiveness of the algorithm. According to experimental results based on the BCI Competition IV Dataset2a, the average classification accuracy is 97.61%. A comparison of the proposed algorithm with other algorithms shows that it performs better in classification. The algorithm can be used to improve the classification performance of MI-based BCIs and BCI systems created for people with disabilities.

A review of critical challenges in MI-BCI: From conventional to deep learning methods

Brain-computer interfaces (BCIs) have achieved significant success in controlling external devices through the Electroencephalogram (EEG) signal processing. BCI-based Motor Imagery (MI) system bridges brain and external devices as communication tools to control, for example, wheelchair for people with disabilities, robotic control, and exoskeleton control. This success largely depends on the machine learning (ML) approaches like deep learning (DL) models. DL algorithms provide effective and powerful models to analyze compact and complex EEG data optimally for MI-BCI applications. DL models with CNN network have revolutionized computer vision through end-to-end learning from raw data. Meanwhile, RNN networks have been able to decode EEG signals by processing sequences of time series data. However, many challenges in the MI-BCI field have affected the performance of DL models. A major challenge is the individual differences in the EEG signal of different subjects. Therefore, the model must be retrained from the scratch for each new subject, which leads to computational costs. Analyzing the EEG signals is challenging due to its low signal to noise ratio and non-stationary nature. Additionally, limited size of existence datasets can lead to overfitting which can be prevented by using transfer learning (TF) approaches. The main contributions of this study are discovering major challenges in the MI-BCI field by reviewing the state of art machine learning models and then suggesting solutions to address these challenges by focusing on feature selection, feature extraction and classification methods.

Transformed common spatial pattern for motor imagery-based brain-computer interfaces

Objective

The motor imagery (MI)-based brain-computer interface (BCI) is one of the most popular BCI paradigms. Common spatial pattern (CSP) is an effective algorithm for decoding MI-related electroencephalogram (EEG) patterns. However, it highly depends on the selection of EEG frequency bands. To address this problem, previous researchers often used a filter bank to decompose EEG signals into multiple frequency bands before applying the traditional CSP.

Approach

This study proposed a novel method, i.e., transformed common spatial pattern (tCSP), to extract the discriminant EEG features from multiple frequency bands after but not before CSP. To verify its effectiveness, we tested tCSP on a dataset collected by our team and a public dataset from BCI competition III. We also performed an online evaluation of the proposed method.

Main results

As a result, for the dataset collected by our team, the classification accuracy of tCSP was significantly higher than CSP by about 8% and filter bank CSP (FBCSP) by about 4.5%. The combination of tCSP and CSP further improved the system performance with an average accuracy of 84.77% and a peak accuracy of 100%. For dataset IVa in BCI competition III, the combination method got an average accuracy of 94.55%, which performed best among all the presented CSP-based methods. In the online evaluation, tCSP and the combination method achieved an average accuracy of 80.00 and 84.00%, respectively.

Significance

The results demonstrate that the frequency band selection after CSP is better than before for MI-based BCIs. This study provides a promising approach for decoding MI EEG patterns, which is significant for the development of BCIs.

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Motor imagery-based brain-computer interfaces (MI-BCI) have important application values in the field of neurorehabilitation and robot control. At present, MI-BCI mostly use bilateral upper limb motor tasks, but there are relatively few studies on single upper limb MI tasks. In this work, we conducted studies on the recognition of motor imagery EEG signals of the right upper limb and proposed a multi-branch fusion convolutional neural network (MF-CNN) for learning the features of the raw EEG signals as well as the two-dimensional time-frequency maps at the same time. The dataset used in this study contained three types of motor imagery tasks: extending the arm, rotating the wrist, and grasping the object, 25 subjects were included. In the binary classification experiment between the grasping object and the arm-extending tasks, MF-CNN achieved an average classification accuracy of 78.52% and kappa value of 0.57. When all three tasks were used for classification, the accuracy and kappa value were 57.06% and 0.36, respectively. The comparison results showed that the classification performance of MF-CNN is higher than that of single CNN branch algorithms in both binary-class and three-class classification. In conclusion, MF-CNN makes full use of the time-domain and frequency-domain features of EEG, can improve the decoding accuracy of single limb motor imagery tasks, and it contributes to the application of MI-BCI in motor function rehabilitation training after stroke.

A novel ANN adaptive Riemannian-based kernel classification for motor imagery

More recently, a number of studies show the interest of the use of the Riemannian geometry in EEG classification. The idea is to exploit the EEG covariance matrices, instead of the raw EEG data, and use the Riemannian geometry to directly classify these matrices. This paper presents a novel Artificial Neural Network approach based on an Adaptive Riemannian Kernel, named ARK-ANN, to classify Electroencephalographic (EEG) motor imaging signals in the context of Brain Computer Interface (BCI). A multilayer perceptron is used to classify the covariance matrices of Motor Imagery (MI) signals employing an adaptive optimization of the testing set. The contribution of a geodesic filter is also assessed for the ANN and the original method which uses an SVM classifier. The results demonstrate that the ARK-ANN performs better than the other methods and the geodesic filter gives slightly better results in the ARK-SVM, considered here as the reference method, in the case of inter-subject classification (accuracy of 87.4% and 86% for ARK-ANN and ARK-SVM, respectively). Regarding the cross-subject classification, the proposed method gives an accuracy of 77.3% and increases the precision by 8.2% in comparison to the SVM based method.

Hybrid fuzzy deep neural network toward temporal-spatial-frequency features learning of motor imagery signals

Achieving an efficient and reliable method is essential to interpret a user's brain wave and deliver an accurate response in biomedical signal processing. However, EEG patterns exhibit high variability across time and uncertainty due to noise and it is a significant problem to be addressed in mental task as motor imagery. Therefore, fuzzy components may help to enable a higher tolerance to noisy conditions. With the advent of Deep Learning and its considerable contributions to Artificial intelligence and data analysis, numerous efforts have been made to evaluate and analyze brain signals. In this study, to make use of neural activity phenomena, the feature extraction preprocessing is applied based on Multi-scale filter bank CSP. In the following, the hybrid series architecture named EEG-CLFCNet is proposed which extract the frequency and spatial features by Compact-CNN and the temporal features by the LSTM network. However, the classification results are evaluated by merging the fully connected network and fuzzy neural block. Here, the proposed method is further validated by the BCI competition IV-2a dataset and compare with two hyperparameter tuning methods, Coordinate-descent and Bayesian optimization algorithm. The proposed architecture that used fuzzy neural block and Bayesian optimization as tuning approach, results in better classification accuracy compared with the state-of-the-art literatures. As results shown, the remarkable performance of the proposed model, EEG-CLFCNet, and the general integration of fuzzy units to other classifiers would pave the way for enhanced MI-based BCI systems.

Brain-Computer Interface using neural network and temporal-spectral features

Brain-Computer Interfaces (BCIs) are increasingly useful for control. Such BCIs can be used to assist individuals who lost mobility or control over their limbs, for recreational purposes such as gaming or semi-autonomous driving, or as an interface toward man-machine integration. Thus far, the performance of algorithms used for thought decoding has been limited. We show that by extracting temporal and spectral features from electroencephalography (EEG) signals and, following, using deep learning neural network to classify those features, one can significantly improve the performance of BCIs in predicting which motor action was imagined by a subject. Our movement prediction algorithm uses Sequential Backward Selection technique to jointly choose temporal and spectral features and a radial basis function neural network for the classification. The method shows an average performance increase of 3.50% compared to state-of-the-art benchmark algorithms. Using two popular public datasets our algorithm reaches 90.08% accuracy (compared to an average benchmark of 79.99%) on the first dataset and 88.74% (average benchmark: 82.01%) on the second dataset. Given the high variability within- and across-subjects in EEG-based action decoding, we suggest that using features from multiple modalities along with neural network classification protocol is likely to increase the performance of BCIs across various tasks.

Subject-Transfer Decoding using the Convolutional Neural Network for Motor Imagery-based Brain-Computer Interface

Various pattern-recognition or machine learning-based methods have recently been developed to improve the accuracy of the motor imagery (MI)-based brain-computer interface (BCI). However, more research is needed to reduce the training time to apply it to the real-world environment. In this study, we propose a subject-transfer decoding method based on a convolutional neural network (CNN) which is robust even with a small number of training trials. The proposed CNN was pre-trained with other subjects' MI data and then fine-tuned to the target subject's training MI data. We evaluated the proposed method using the BCI competition IV data2a, which had the 4-class MIs. Consequently, on the same test dataset, with changing the number of training trials, the proposed method showed better accuracy than the self-training method, which used only the target subject's data for training, as averaged 86.54±7.78% (288 trials), 85.76 ±8.00% (240 trials), 84.65±8.11% (192 trials), and 83.29 ±8.25% (144 trials), respectively, which was 4.94% (288 trials), 6.10% (240 trials), 9.03% (192 trials), and 12.31% (144 trials)-point higher than the self-training method. Consequently, the proposed method was shown to be effective in maintaining classification accuracy even with the reduced training trials.

How to successfully classify EEG in motor imagery BCI: a metrological analysis of the state of the art

Objective. Processing strategies are analysed with respect to the classification of electroencephalographic signals related to brain-computer interfaces based on motor imagery. A review of literature is carried out to understand the achievements in motor imagery classification, the most promising trends, and the challenges in replicating these results. Main focus is placed on performance by means of a rigorous metrological analysis carried out in compliance with the international vocabulary of metrology. Hence, classification accuracy and its uncertainty are considered, as well as repeatability and reproducibility.Approach. The paper works included in the review concern the classification of electroencephalographic signals in motor-imagery- based brain-computer interfaces. Article search was carried out in accordance with the PRISMA standard and 89 studies were included.Main results. Statistically-based analyses show that brain-inspired approaches are increasingly proposed, and that these are particularly successful in discriminating against multiple classes. Notably, many proposals involve convolutional neural networks. Instead, classical machine learning approaches are still effective for binary classifications. Many proposals combine common spatial pattern, least absolute shrinkage and selection operator, and support vector machines. Regarding reported classification accuracies, performance above the upper quartile is in the 85 % to 100 % range for the binary case and in the 83 % to 93 % range for multi-class one. Associated uncertainties are up to 6 % while repeatability for a predetermined dataset is up to 8 %. Reproducibility assessment was instead prevented by lack of standardization in experiments.Significance. By relying on the analysed studies, the reader is guided towards the development of a successful processing strategy as a crucial part of a brain-computer interface. Moreover, it is suggested that future studies should extend these approaches on data from more subjects and with custom experiments, even by investigating online operation. This would also enable the quantification of results reproducibility.

Improved Brain-Computer Interface Signal Recognition Algorithm Based on Few-Channel Motor Imagery

Common spatial pattern (CSP) is an effective algorithm for extracting electroencephalogram (EEG) features of motor imagery (MI); however, CSP mainly aims at multichannel EEG signals, and its effect in extracting EEG features with fewer channels is poor-even worse than before using CSP. To solve the above problem, a new combined feature extraction method has been proposed in this study. For EEG signals from fewer channels (three channels), wavelet packet transform, fast ensemble empirical mode decomposition, and local mean decomposition were used to decompose the band-pass filtered EEG into multiple time-frequency components, and the corresponding components were selected according to the frequency characteristics of MI or the correlation coefficient between its time-frequency components and the original EEG signal. Furthermore, phase space reconstruction (PSR) was performed on the selected components after the three time-frequency decompositions, the maximum Lyapunov index was calculated, and the features were reconstructed; then, CSP projection mapping was used for the reconstructed features. The support vector machine probability output model was trained by the obtained three mappings. Probability outputs by three different support vector machines were then obtained. Finally, the classification of test samples was determined by the fusion of the Dempster-Shafer evidence theory at the decision level. The results showed that the accuracy of the proposed method was 95.71% on data set III of BCI competition II (left- and right-hand MI), which was 2.88% higher than the existing methods. On data set IIb of BCI competition IV, the average accuracy was 86.60%, which was 2.3% higher than the existing methods. This study verified the effectiveness of the proposed method and provided an approach for the research and development of the MI-BCI system based on fewer channels.

A transfer learning-based CNN and LSTM hybrid deep learning model to classify motor imagery EEG signals

In the Motor Imagery (MI)-based Brain Computer Interface (BCI), users' intention is converted into a control signal through processing a specific pattern in brain signals reflecting motor characteristics. There are such restrictions as the limited size of the existing datasets and low signal to noise ratio in the classification of MI Electroencephalogram (EEG) signals. Machine learning (ML) methods, particularly Deep Learning (DL), have overcome these limitations relatively. In this study, three hybrid models were proposed to classify the EEG signal in the MI-based BCI. The proposed hybrid models consist of the convolutional neural networks (CNN) and the Long-Short Term Memory (LSTM). In the first model, the CNN with different number of convolutional-pooling blocks (from shallow to deep CNN) was examined; a two-block CNN model not affected by the vanishing gradient descent and yet able to extract desirable features employed; the second and third models contained pre-trained CNNs conducing to the exploration of more complex features. The transfer learning strategy and data augmentation methods were applied to overcome the limited size of the datasets by transferring learning from one model to another. This was achieved by employing two powerful pre-trained convolutional neural networks namely ResNet-50 and Inception-v3. The continuous wavelet transform (CWT) was used to generate images for the CNN. The performance of the proposed models was evaluated on the BCI Competition IV dataset 2a. The mean accuracy vlaues of 86%, 90%, and 92%, and mean Kappa values of 81%, 86%, and 88% were obtained for the hybrid neural network with the customized CNN, the hybrid neural network with ResNet-50 and the hybrid neural network with Inception-v3, respectively. Despite the promising performance of the three proposed models, the hybrid neural network with Inception-v3 outperformed the two other models. The best obtained result in the present study improved the previous best result in the literature by 7% in terms of classification accuracy. From the findings, it can be concluded that transfer learning based on a pre-trained CNN in combination with LSTM is a novel method in MI-based BCI. The study also has implications for the discrimination of motor imagery tasks in each EEG recording channel and in different brain regions which can reduce computational time in future works by only selecting the most effective channels.

Decoding ECoG signal into 3D hand translation using deep learning

Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.

A Two-Branch CNN Fusing Temporal and Frequency Features for Motor Imagery EEG Decoding

With the development of technology and the rise of the meta-universe concept, the brain-computer interface (BCI) has become a hotspot in the research field, and the BCI based on motor imagery (MI) EEG has been widely concerned. However, in the process of MI-EEG decoding, the performance of the decoding model needs to be improved. At present, most MI-EEG decoding methods based on deep learning cannot make full use of the temporal and frequency features of EEG data, which leads to a low accuracy of MI-EEG decoding. To address this issue, this paper proposes a two-branch convolutional neural network (TBTF-CNN) that can simultaneously learn the temporal and frequency features of EEG data. The structure of EEG data is reconstructed to simplify the spatio-temporal convolution process of CNN, and continuous wavelet transform is used to express the time-frequency features of EEG data. TBTF-CNN fuses the features learned from the two branches and then inputs them into the classifier to decode the MI-EEG. The experimental results on the BCI competition IV 2b dataset show that the proposed model achieves an average classification accuracy of 81.3% and a kappa value of 0.63. Compared with other methods, TBTF-CNN achieves a better performance in MI-EEG decoding. The proposed method can make full use of the temporal and frequency features of EEG data and can improve the decoding accuracy of MI-EEG.

Time-Distributed Attention Network for EEG-based Motor Imagery Decoding from the Same Limb

A brain-computer interface (BCI) based on motor imagery (MI) from the same limb can provide an intuitive control pathway but has received limited attention. It is still a challenge to classify multiple MI tasks from the same limb. The goal of this study is to propose a novel decoding method to classify the MI tasks of four joints of the same upper limb and the resting state. EEG signals were collected from 20 participants. A time-distributed attention network (TD-Atten) was proposed to adaptively assign different weights to different classes and frequency bands of the input multiband Common Spatial Pattern (CSP) features. The long short-term memory (LSTM) and dense layers were then used to learn sequential information from the reweight features and perform the classification. Our proposed method outperformed other baseline and deep learning-based methods and obtained the accuracies of 46.8% in the 5-class scenario and 53.4% in the 4-class scenario. The visualization results of attention weights indicated that the proposed framework can adaptively pay attention to alpha-band related features in MI tasks, which was consistent with the analysis of brain activation patterns. These results demonstrated the feasibility and interpretability of the attention mechanism in MI decoding and the potential of this fine MI paradigm to be applied for the control of a robotic arm or a neural prosthesis.

Functional Reorganization After Four-Week Brain-Computer Interface-Controlled Supernumerary Robotic Finger Training: A Pilot Study of Longitudinal Resting-State fMRI

Humans have long been fascinated by the opportunities afforded through motor augmentation provided by the supernumerary robotic fingers (SRFs) and limbs (SRLs). However, the neuroplasticity mechanism induced by the motor augmentation equipment still needs further investigation. This study focused on the resting-state brain functional reorganization during longitudinal brain-computer interface (BCI)-controlled SRF training in using the fractional amplitude of low-frequency fluctuation (fALFF), regional homogeneity (ReHo), and degree centrality (DC) metrics. Ten right-handed subjects were enrolled for 4 weeks of BCI-controlled SRF training. The behavioral data and the neurological changes were recorded at baseline, training for 2 weeks, training for 4 weeks immediately after, and 2 weeks after the end of training. One-way repeated-measure ANOVA was used to investigate long-term motor improvement [F(2.805,25.24) = 43.94, p < 0.0001] and neurological changes. The fALFF values were significantly modulated in Cerebelum_6_R and correlated with motor function improvement (r = 0.6887, p < 0.0402) from t0 to t2. Besides, Cerebelum_9_R and Vermis_3 were also significantly modulated and showed different trends in longitudinal SRF training in using ReHo metric. At the same time, ReHo values that changed from t0 to t1 in Vermis_3 was significantly correlated with motor function improvement (r = 0.7038, p < 0.0344). We conclude that the compensation and suppression mechanism of the cerebellum existed during BCI-controlled SRF training, and this current result provided evidence to the neuroplasticity mechanism brought by the BCI-controlled motor-augmentation devices.

Deep Feature Mining via the Attention-Based Bidirectional Long Short Term Memory Graph Convolutional Neural Network for Human Motor Imagery Recognition

Recognition accuracy and response time are both critically essential ahead of building the practical electroencephalography (EEG)-based brain–computer interface (BCI). However, recent approaches have compromised either the classification accuracy or the responding time. This paper presents a novel deep learning approach designed toward both remarkably accurate and responsive motor imagery (MI) recognition based on scalp EEG. Bidirectional long short-term memory (BiLSTM) with the attention mechanism is employed, and the graph convolutional neural network (GCN) promotes the decoding performance by cooperating with the topological structure of features, which are estimated from the overall data. Particularly, this method is trained and tested on the short EEG recording with only 0.4 s in length, and the result has shown effective and efficient prediction based on individual and groupwise training, with 98.81% and 94.64% accuracy, respectively, which outperformed all the state-of-the-art studies. The introduced deep feature mining approach can precisely recognize human motion intents from raw and almost-instant EEG signals, which paves the road to translate the EEG-based MI recognition to practical BCI systems.

Accurate prediction and further dissection of neonicotinoid elimination in the water treatment by CTS@AgBC using multihead attention-based convolutional neural network combined with the time-dependent Cox regression model

Imidacloprid (IMI), as the most widely used neonicotinoid insecticide, poses a serious threat to the water ecosystem due to the inefficient elimination in the traditional water treatment. Chitosan (CTS)-stabilized biochar (BC)-supported Ag nanoparticles (CTS@AgBC) are applied to eliminate the IMI in the water treatment effectively. Batch experiments depict that the modification of BC by CTS and Ag nanoparticles remarkably improve its adsorption performance. The pseudo-second-order and Elovich models have good performance in simulating the adsorption processes of CTS@AgBC and BC. This indicates that the chemical adsorption on real surfaces plays the dominant role in the adsorption of IMI by CTS@AgBC and BC. In addition, the multihead attention (MHA)-based convolutional neural network (CNN) combined with the time-dependent Cox regression model are initially applied to predict and dissect the adsorption elimination processes of IMI by CTS@AgBC. The proposed MHA-CNN model achieves more accurate concentration prediction of IMI than traditional models. According to influence weights by MHA module, biochar category, pH, and treatment temperature are considered the three dominant environmental variables to determine the IMI elimination processes. This study provides insights into roles of environmental variables in the elimination of IMI by CTS@AgBC and the accurate prediction of IMI concentration.

A multiscale siamese convolutional neural network with cross-channel fusion for motor imagery decoding

BACKGROUND

Recently, convolutional neural networks (CNN) are widely applied in motor imagery electroencephalography (MI-EEG) signal classification tasks. However, a simple CNN framework is challenging to satisfy the complex MI-EEG signal decoding.

NEW METHOD

In this study, we propose a multiscale Siamese convolutional neural network with cross-channel fusion (MSCCF-Net) for MI-EEG classification tasks. The proposed network consists of three parts: Siamese cross-channel fusion streams, similarity module and classification module. Each Siamese cross-channel fusion stream contains multiple branches, and each branch is supplemented by cross-channel fusion modules to improve multiscale temporal feature representation capability. The similarity module is adopted to measure the feature similarity between multiple branches. At the same time, the classification module provides a strong constraint to classify the features from all Siamese cross-channel fusion streams. The combination of the similarity module and classification module constitutes a new joint training strategy to further optimize the network performance.

RESULTS

The experiment is conducted on the public BCI Competition IV 2a and 2b datasets, and the results show that the proposed network achieves an average accuracy of 87.36% and 87.33%, respectively.

COMPARISON WITH EXISTING METHODS AND CONCLUSIONS

The proposed network adopts cross-channel fusion to learn multiscale temporal characteristics and joint training strategy to optimize the training process. Therefore, the performance outperforms other state-of-the-art MI-EEG signal classification methods.

Feature Extraction Method Based on Filter Banks and Riemannian Tangent Space in Motor-Imagery BCI

Optimal feature extraction for multi-category motor imagery brain-computer interfaces (MI-BCIs) is a research hotspot. The common spatial pattern (CSP) algorithm is one of the most widely used methods in MI-BCIs. However, its performance is adversely affected by variance in the operational frequency band and noise interference. Furthermore, the performance of CSP is not satisfactory when addressing multi-category classification problems. In this work, we propose a fusion method combining Filter Banks and Riemannian Tangent Space (FBRTS) in multiple time windows. FBRTS uses multiple filter banks to overcome the problem of variance in the operational frequency band. It also applies the Riemannian method to the covariance matrix extracted by the spatial filter to obtain more robust features in order to overcome the problem of noise interference. In addition, we use a One-Versus-Rest support vector machine (OVR-SVM) model to classify multi-category features. We evaluate our FBRTS method using BCI competition IV dataset 2a and 2b. The experimental results show that the average classification accuracy of our FBRTS method is 77.7% and 86.9% in datasets 2a and 2b respectively. By analyzing the influence of the different numbers of filter banks and time windows on the performance of our FBRTS method, we can identify the optimal number of filter banks and time windows. Additionally, our FBRTS method can obtain more distinctive features than the filter banks common spatial pattern (FBCSP) method in two-dimensional embedding space. These results show that our proposed method can improve the performance of MI-BCIs.

A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery based Brain-Computer Interface

The motor imagery (MI) based brain-computer interfaces (BCIs) have been proposed as a potential physical rehabilitation technology. However, the low classification accuracy achievable with MI tasks is still a challenge when building effective BCI systems. We propose a novel MI classification model based on measurement of functional connectivity between brain regions and graph theory. Specifically, motifs describing local network structures in the brain are extracted from functional connectivity graphs. A graph embedding model called Ego-CNNs is then used to build a classifier, which can convert the graph from a structural representation to a fixed-dimensional vector for detecting critical structure in the graph. We validate our proposed method on four datasets, and the results show that our proposed method produces high classification accuracies in two-class classification tasks (92.8% for dataset 1, 93.4% for dataset 2, 96.5% for dataset 3, and 80.2% for dataset 4) and multiclass classification tasks (90.33% for dataset 1). Our proposed method achieves a mean Kappa value of 0.88 across nine participants, which is superior to other methods we compared it to. These results indicate that there is a local structural difference in functional connectivity graphs extracted under different motor imagery tasks. Our proposed method has great potential for motor imagery classification in future studies.

Knowledge-driven feature component interpretable network for motor imagery classification

OBJECTIVE

End-to-end convolution neural network (CNN) has achieved great success in motor imagery classification without manual feature design. However, all the existing deep network solutions are purely data-driven and lack interpretability, which makes it impossible to discover insightful knowledge from the learnt features, not to mention to design specific network structure. The heavy computational cost of CNN also makes it challenging for real time application along with high classification performance.

APPROACH

To address these problems, a novel Knowledge-driven Feature Component Interpretable Network (KFCNet) was proposed, which combines spatial and temporal convolution in analogy to ICA and power spectrum pipeline. Prior frequency band knowledge of sensory motor rhythms (SMR) has been formulated as band-pass linear-phase digit FIR filters to initialize the temporal convolution kernels to enable knowledge driven mechanism. To avoid signal distortion and achieve linear phase and unimodality of filters, a symmetry loss is proposed, which is used in combination with the cross-entropy classification loss for training. Besides the general prior knowledge, subject specific time-frequency property of ERDS (event-related desynchronization and synchronization) has been employed to construct and initialize the network with significantly fewer parameters.

MAIN RESULTS

Comparison experiments on two public datasets have been performed. Interpretable feature components could be observed in the trained model. The physically meaningful observation could efficiently assist the network structure design. Excellent classification performance on motor imagery has been obtained.

SIGNIFICANCE

The performance of KFCNet is comparative to the state-of-the-art methods but with much fewer parameters and makes real time application possible.

Evaluating Convolutional Neural Networks as a Method of EEG-EMG Fusion

Wearable robotic exoskeletons have emerged as an exciting new treatment tool for disorders affecting mobility; however, the human–machine interface, used by the patient for device control, requires further improvement before robotic assistance and rehabilitation can be widely adopted. One method, made possible through advancements in machine learning technology, is the use of bioelectrical signals, such as electroencephalography (EEG) and electromyography (EMG), to classify the user's actions and intentions. While classification using these signals has been demonstrated for many relevant control tasks, such as motion intention detection and gesture recognition, challenges in decoding the bioelectrical signals have caused researchers to seek methods for improving the accuracy of these models. One such method is the use of EEG–EMG fusion, creating a classification model that decodes information from both EEG and EMG signals simultaneously to increase the amount of available information. So far, EEG–EMG fusion has been implemented using traditional machine learning methods that rely on manual feature extraction; however, new machine learning methods have emerged that can automatically extract relevant information from a dataset, which may prove beneficial during EEG–EMG fusion. In this study, Convolutional Neural Network (CNN) models were developed using combined EEG–EMG inputs to determine if they have potential as a method of EEG–EMG fusion that automatically extracts relevant information from both signals simultaneously. EEG and EMG signals were recorded during elbow flexion–extension and used to develop CNN models based on time–frequency (spectrogram) and time (filtered signal) domain image inputs. The results show a mean accuracy of 80.51 ± 8.07% for a three-class output (33.33% chance level), with an F-score of 80.74%, using time–frequency domain-based models. This work demonstrates the viability of CNNs as a new method of EEG–EMG fusion and evaluates different signal representations to determine the best implementation of a combined EEG–EMG CNN. It leverages modern machine learning methods to advance EEG–EMG fusion, which will ultimately lead to improvements in the usability of wearable robotic exoskeletons.